You can create /var/adm/loginlog with permissions 0600. If the login failed
it is logged in this file after 'n' wrong user-IDs/passwords (n=5 per
default, you may change this with RETRIES=x in /etc/default/login to x). You
can write a script to parse /var/adm/loginlog. Notice that this behaviour
you try to implement can be used for a DoS attack and is not restricted to
telnet. Your users should use ssh anyway to prevent package sniffer attacks.
Hope this helps.