The feature is patterned after the UNIX remote access control feature, so they used the same "hosts.allow" name for the file. The one in /etc will be the one for UNIX. The ASE one goes in $SYBASE by default, thought yo can configure backupserver to have it somewhere else. I don't think the install process creates the file by default.
This is kind of a shotgun approach to getting past the error, but try creating a file named "hosts.allow" in $SYBASE with the content
physical1.example.com +
physical2.example.com +
virtual.example.com +
172.xx.xx.1 +
172.xx.xx.2 +
172.xx.xx.0 +
and then retry installing backupserver.
If that doesn't work: It is pretty easy to create the backpserver manually (it is just a matter of creating an interfaces file entry with the "dsedit" utility, creating a RUN_servername_BS file in $SYBASE/ASE-15_0/install, and fixing up the sysservers.srvnetname value for the SYB_BACKUP entry in the ASE. You may get the same error when trying to dump; it may be the error only happens on one side of the HA setup.
Here is a copy of one of my RUN files, just substitute in your $SYBASE value for my "/work/releases/rel1572.6" and the name of your server for my "rel15726_brets_sun_back":
bret-sun2% cat RUN_rel15726_brets_sun_back
#!/bin/sh
#
# Backup Server name: rel15726_brets_sun_back
# Maximum number of network connections: 25
# Maximum number of server connections: 20
# Multibuf executable path: /work/releases/rel1572.6/ASE-15_0/bin/sybmultbuf
#
/work/releases/rel1572.6/ASE-15_0/bin/backupserver \
-Srel15726_brets_sun_back \
-e/work/releases/rel1572.6/ASE-15_0/install/rel15726_brets_sun_back.log \
-N25 \
-C20 \
-M/work/releases/rel1572.6/ASE-15_0/bin/sybmultbuf \
In the ASE:
use master
go
sp_configure "allow updates", 1
go
update sysservers set srvnetname = <your_bs_servername> where srvname = "SYB_BACKUP"
go
sp_configure "allow updates", 0
go
-bret