UNIX -->EBCIDIC

UNIX -->EBCIDIC

Post by Neil Ricke » Sat, 26 Sep 1998 04:00:00




>I'm trying to automate a ftp process that send a file from
>a Sun (maybe OS or solaris) sever to an IBM mainframe.
>And need to conversion from ASCII to EBCIDIC before sending the file up.
>What are my options re: scripts, utils programs .

The ftp server on IBM mainframes will ordinarily convert ASCII files
to EBCDIC while uploading them.  If you specify binary, they will
not convert.

It is also possible to convert within a unix system, using the
'dd' command (check the man pages).

Probably the best way to automate 'ftp' is with 'expect' scripts, if
you have 'expect' installed on your unix system.  If the task is
simple enough, you might be able to create a special user account to
do the transfer, and put much of the automation in the '.netrc'
file.  The man pages for 'ftp' should explain how to use this file.

 
 
 

UNIX -->EBCIDIC

Post by Nick Maclar » Sun, 27 Sep 1998 04:00:00




Quote:>i thought the way it worked was that if you specify 'ascii', then the ftp on
>your box put your ascii file into a format common to all ftp (telnet format,
>i think), and the receiving box's job was to convert that to whatever
>'ascii' means in it's own world. that is what makes ftp so universally
>used - you don't have to guess who was on the other end of the wire.

This is seriously misleading.  There are far more differences between
IBM MVS and Unix than just the character set, and no FTP implementation
that I know of gets them right.  When I was coordinating data transfer
from MVS to Unix, I wrote some (MVS) utilities that did, but they did
assume a C implementation and called other services that you don't have.

Yes, this is the best way for most users to transfer files, but you
CAN'T avoid knowing what system you are transferring to, and munging
round the problems accordingly.  This applies even Unix-Unix, by the
way - but you are unlikely to hit the problems in that case.

If anyone wants the source, please ask me, but doing the job competently
needs a fairly good knowledge of both MVS and Unix data management.

Quote:>i thought if you use the dd option to convert ascii to ebcdic, then you just
>did what ftp would have done anyway. (if you use dd to do blocking, etc.,
>that's another matter).

That is completely wrong.  dd's fancier facilities are, and always
have been, an absolute disaster.  I first attempted to use it for
(a) that purpose and (b) tape handling back in 1979, and decided
that it was much worse than useless.  It HAS improved, but I don't
think this area has.

For example, it uses a version of the conversion table that is
unique to itself - it is unlike any that I have ever seen, and I
have a lot of experience with this problem.  It gets the handling
of control characters, trailing spaces and other MVS/Unix problems
completely wrong.  And so on.

And, please note that many of these errors are because of its DESIGN,
rather than bugs in the code.  That is one reason that I am pretty
certain they will still be there, though I haven't checked them in
several years.

DON'T USE DD FOR EITHER MVS/UNIX CONVERSION OR TAPE HANDLING.

Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QG, England.

Tel.:  +44 1223 334761    Fax:  +44 1223 334679

 
 
 

UNIX -->EBCIDIC

Post by Gerry Sinkiewi » Sun, 27 Sep 1998 04:00:00


Where I work we send JCL via ftp (type=JES) to an internal reader (MVS talk).
That JCL then performs an FTP get with the proper settings so that MVS is happy.
This means that someone needs to prepare a cshell that writes the JCL properly.

As is said MVS can be particular, by using JCL perparted by MVS knowledgeable
people the MVS end works fine. Typically the cshell needs to alter just a few
names via $arg[1] etc.





>>i thought the way it worked was that if you specify 'ascii', then the ftp on
>>your box put your ascii file into a format common to all ftp (telnet format,
>>i think), and the receiving box's job was to convert that to whatever
>>'ascii' means in it's own world. that is what makes ftp so universally
>>used - you don't have to guess who was on the other end of the wire.

>This is seriously misleading.  There are far more differences between
>IBM MVS and Unix than just the character set, and no FTP implementation
>that I know of gets them right.  When I was coordinating data transfer
>from MVS to Unix, I wrote some (MVS) utilities that did, but they did
>assume a C implementation and called other services that you don't have.

>Yes, this is the best way for most users to transfer files, but you
>CAN'T avoid knowing what system you are transferring to, and munging
>round the problems accordingly.  This applies even Unix-Unix, by the
>way - but you are unlikely to hit the problems in that case.

>If anyone wants the source, please ask me, but doing the job competently
>needs a fairly good knowledge of both MVS and Unix data management.

>>i thought if you use the dd option to convert ascii to ebcdic, then you just
>>did what ftp would have done anyway. (if you use dd to do blocking, etc.,
>>that's another matter).

>That is completely wrong.  dd's fancier facilities are, and always
>have been, an absolute disaster.  I first attempted to use it for
>(a) that purpose and (b) tape handling back in 1979, and decided
>that it was much worse than useless.  It HAS improved, but I don't
>think this area has.

>For example, it uses a version of the conversion table that is
>unique to itself - it is unlike any that I have ever seen, and I
>have a lot of experience with this problem.  It gets the handling
>of control characters, trailing spaces and other MVS/Unix problems
>completely wrong.  And so on.

>And, please note that many of these errors are because of its DESIGN,
>rather than bugs in the code.  That is one reason that I am pretty
>certain they will still be there, though I haven't checked them in
>several years.

>DON'T USE DD FOR EITHER MVS/UNIX CONVERSION OR TAPE HANDLING.

>Regards,
>Nick Maclaren,
>University of Cambridge Computing Service,
>New Museums Site, Pembroke Street, Cambridge CB2 3QG, England.

>Tel.:  +44 1223 334761    Fax:  +44 1223 334679

Gerald P. Sinkiewicz

 
 
 

UNIX -->EBCIDIC

Post by Arran Pric » Tue, 29 Sep 1998 04:00:00


<snip

Quote:> >man dd

> I'm not sure that DJ even needs dd. FTP on IBM mainframe (both client and
> server) understand that ASCII means translate to EBCDIC before storing. Try
> this: treat the file as a text file for the purposes of your FTP (transfer
> mode ASCII), and send the file to the mainframe. I'll wager half a byte
> (4 bits) that it shows up as an EBCDIC file on the mainframe.

> Lew Pitcher
> Joat-in-training

yep sounds right...
one thing to watch tho is that you dont have packed fields etc in your
data....

Arran
My opinions are my own and do not reflect those of my employer.

 
 
 

UNIX -->EBCIDIC

Post by James W. Ada » Tue, 29 Sep 1998 04:00:00


Quote:> I'm not sure that DJ even needs dd. FTP on IBM mainframe (both client and
> server) understand that ASCII means translate to EBCDIC before storing. Try
> this: treat the file as a text file for the purposes of your FTP (transfer
> mode ASCII), and send the file to the mainframe. I'll wager half a byte
> (4 bits) that it shows up as an EBCDIC file on the mainframe.

To cut to the chase here, the facts are:

MVS has a record structured filesystem.  This means that files are
organized into records of fixed or variable size, and blocked into
control intervals.  UNIX files are streams of bytes.  Thus, anyone
transferring files from a UNIX system to a MVS system needs to be
aware of this.  While the MVS TCP suite will try to catalog and
allocate (separate operations) a file using default parameters,
it is often necessary for the user on a UNIX box to send quote SITE
commands to the MVS FTP server to set filesystem parameters which
have no UNIX equivalents.  Indeed, the MVS FTP server will explain
these if you ask it politely.  Log in from UNIX and send the command:
quote help site
Also, sending some types of variable-length-record files from MVS
to UNIX will create gibberish because there is no UNIX facility to
provide record delimiters other than inserting newlines.

EBCDIC contains characters which have no ASCII equivalents, and
vice versa.  There is no universally agreed upon way to translate
one into the other.

Of course, binary files (including packed record formats on MVS)
are inherently nonportable.  Remember that the MVS filesystem does
lots of things behind the scenes that the UNIX filesystem doesn't,
and instead relies on applications to do.  Just because a file
looks like text in a TSO session doesn't mean you can simply
FTP it across in ASCII mode.

Hence, the key is data portability at the application level.  If you
just want to send 80-column fixed-length card image text files back and
forth for applications development, you may be OK just using FTP in
ASCII mode, so long as you avoid characters which lie outside
common alphanumeric text and common punctuation.

Otherwise, you need to convert files to some portable format which
avoids dependencies on things like word size, byte ordering,
filesystem structure, character set and collating sequence.

Those of us who have been around UNIX for a long time tend to
be more conscious of data portability, but we still make assumptions
which may not be valid on other platforms.

It is *impossible* to simply "translate" between MVS EBCDIC and UNIX
ASCII.  They are partially intersecting sets.

With these factors in mind, it is almost always possible to "translate"
between them for a specific purpose, but never possible to do so in
the general case.

Both MVS and UNIX provide a host of utilities to translate data.
Going between them will usually require some combination of both,
as well as a thorough understanding of their differences, and also
the requirements of the consumer applications.

--

        "I became obsessed with angels and ballerinas,
         things of grace and beauty, otherworldly."
C*tesville, VA  22903         --C. Love

 
 
 

UNIX -->EBCIDIC

Post by Lew Pitch » Wed, 30 Sep 1998 04:00:00


Agreed, my suggestion wasn't perfect, but for the same reasons, no
suggestion will be.

If our friend has to transfer text files from an ASCII based unix system to
an EBCDIC based IBM mainframe, he will have characterset translation and
record image problems no matter what tool he uses.

No matter what method of data transfer is used, our friend will have to be
aware of and plan for both an imperfect character mapping process and an
imperfect record mapping process. So long as he chooses a process which
produces results he can live with, he should be ok. Otherwise, he might as
well either give up (worst case, it cant be done), or look elsewhere.

IBM's FTP is consistant in it's characterset translation and record
mapping. Since perfection is not obtainable, consistancy will have to do.
Can our friend live with the imperfect, but consistant, mappings that IBM's
FTP provides? Only he can answer that.

Lew Pitcher
Joat-in-training

 
 
 

UNIX -->EBCIDIC

Post by Garry Garre » Thu, 01 Oct 1998 04:00:00





> >i thought the way it worked was that if you specify 'ascii', then the ftp on
> >your box put your ascii file into a format common to all ftp (telnet format,
> >i think), and the receiving box's job was to convert that to whatever
> >'ascii' means in it's own world. that is what makes ftp so universally
> >used - you don't have to guess who was on the other end of the wire.

The opposite of "ascii" in FTP is "binary" (which basically means send
the file "as is").  "ascii" doesn't mean quite what you think it does.
There is some agreement as to what "ascii" means, largely because FTP
grew up on unix - bare linefeeds represent new lines, etc.  If you get
beyond 7 bit ascii, your guess is as good as mine as to how it is handled.
Basically, anybody who uses a "non-standard" (and relize that this is
pretty fuzzy) character set (oh like say EBCDIC) is responsible for
doing any required translation.  In the case of MVS, if you are FTPing
in "ascii" mode, then it is your mainframe's FTP daemon (if you are
FTPing into the mainframe) or FTP client (if you are FTPing into unix
from the mainframe) who will translate from EBCDIC to ASCII.

About 5 years ago (at a previous job) I was knee deep in FTPing files
between unix and MVS.  I found MVS's EBCDIC to ASCII translation table
to be woefully inadequate.  I ended up writing my own and having the
MVS folks install it (they would not make it the standard table, I
had to issue a "quote site" command to pick the translation table that
I wanted).  It's been so long ago, I'm not sure I could even dig up
my old translation table.  The stock standard one (and maybe IBM has
changed this) assumed the orginal PC ascii (and my suns were using
ISO-Latin-1 ascii) when downloading and 7 bit ascii when uploading.
I could not believe it myself - they used a different translation table
for uploading and downloading, and their tables did not have 1:1
translations!

Quote:> >i thought if you use the dd option to convert ascii to ebcdic, then you just
> >did what ftp would have done anyway. (if you use dd to do blocking, etc.,
> >that's another matter).

In theory maybe.  If you have very simple text (a-z,A-Z,0-9) then perhaps
it is equivalent.  The major difference is where the translation happens:
on the mainframe when it is sending the file or on your unix box when you
process the incoming/outgoing file.  If you have very simple text (I missed
the start of this tread), ask yourself where you want to spend the CPU?

I don't know anyone who actually uses dd for very much but very simple
reports.  Everyone I know who is seriously moving data between ASCII and
EBCDIC tries to use dd, and after a month or so, they invariably go out
and write their own translation program.

--
-----------------------------------------------------------------------
Garry J. Garrett              
CSG Systems, Inc.      ._o    see my homepage for a "mailto:" tag
2525 North 117th Ave.    |>   to send me e-mail...
Mailstop 2-A             4    
Omaha, NE 68164-3679          

CSG Systems  - http://www.csgsys.com/
CSG Internal - http://intranet/unixops/
My Homepage  - http://monarch.papillion.ne.us/~ggarrett

I do not speak in any capacity on behalf of CSG Systems.
I get into enough trouble speaking for myself.  

 
 
 

UNIX -->EBCIDIC

Post by Nick Maclar » Fri, 02 Oct 1998 04:00:00






|> > >i thought the way it worked was that if you specify 'ascii', then the ftp on
|> > >your box put your ascii file into a format common to all ftp (telnet format,
|> > >i think), and the receiving box's job was to convert that to whatever
|> > >'ascii' means in it's own world. that is what makes ftp so universally
|> > >used - you don't have to guess who was on the other end of the wire.
|>
|> The opposite of "ascii" in FTP is "binary" (which basically means send
|> the file "as is").  "ascii" doesn't mean quite what you think it does.

Please be a bit more careful with your attributions - you removed
everything that I said, but left my name in.

Also, that is seriously misleading, especially in context.  "binary"
does NOT mean transfer the file "as is", not even remotely.  It means
to transfer it as a stream of untranslated 8-bit bytes, which is in
no sense "as is" for MVS files.  Actually, there are two forms of
"binary" in FTP, but let's skip that horror :-(

|> I don't know anyone who actually uses dd for very much but very simple
|> reports.  Everyone I know who is seriously moving data between ASCII and
|> EBCDIC tries to use dd, and after a month or so, they invariably go out
|> and write their own translation program.

That is definitely so.  IBM's FTP may be a horror, but it is much
bbetter than dd.

I have reasonably good translation tables, if anyone needs them,
for code page 037 and 500, and in 7- and 8-bit forms.

Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QG, England.

Tel.:  +44 1223 334761    Fax:  +44 1223 334679

 
 
 

UNIX -->EBCIDIC

Post by Robert Butra » Wed, 07 Oct 1998 04:00:00


WHAT AM I DOING WRONG?   ;)

When I use ftp from Unix to AS/400 it works perfectly!
I do not specify "ascii" although it is the default send mode.  I DO NOT
specify binary since that precludes any translation and sends the EBCDIC
(AS/400) file to the ASCII machine "as is".  I send files regularly both
ways and my only complaint is the speed 250K give or take.  Of course
"stream" data has always eaten the S/36 and AS/400 machines alive.  They
really prefer record fixed length format...

Maybe I remembered to send files without "packed" numerics...

Robert Butram
http://www.butram.com

The opinion expressed here may NOT reflect
the reality experienced by others





> >i thought the way it worked was that if you specify 'ascii', then the
ftp on
> >your box put your ascii file into a format common to all ftp (telnet
format,
> >i think), and the receiving box's job was to convert that to whatever
> >'ascii' means in it's own world. that is what makes ftp so universally
> >used - you don't have to guess who was on the other end of the wire.

> This is seriously misleading.  There are far more differences between
> IBM MVS and Unix than just the character set, and no FTP implementation
> that I know of gets them right.  When I was coordinating data transfer
> from MVS to Unix, I wrote some (MVS) utilities that did, but they did
> assume a C implementation and called other services that you don't have.

> Yes, this is the best way for most users to transfer files, but you
> CAN'T avoid knowing what system you are transferring to, and munging
> round the problems accordingly.  This applies even Unix-Unix, by the
> way - but you are unlikely to hit the problems in that case.

> If anyone wants the source, please ask me, but doing the job competently
> needs a fairly good knowledge of both MVS and Unix data management.

> >i thought if you use the dd option to convert ascii to ebcdic, then you
just
> >did what ftp would have done anyway. (if you use dd to do blocking,
etc.,
> >that's another matter).

> That is completely wrong.  dd's fancier facilities are, and always
> have been, an absolute disaster.  I first attempted to use it for
> (a) that purpose and (b) tape handling back in 1979, and decided
> that it was much worse than useless.  It HAS improved, but I don't
> think this area has.

> For example, it uses a version of the conversion table that is
> unique to itself - it is unlike any that I have ever seen, and I
> have a lot of experience with this problem.  It gets the handling
> of control characters, trailing spaces and other MVS/Unix problems
> completely wrong.  And so on.

> And, please note that many of these errors are because of its DESIGN,
> rather than bugs in the code.  That is one reason that I am pretty
> certain they will still be there, though I haven't checked them in
> several years.

> DON'T USE DD FOR EITHER MVS/UNIX CONVERSION OR TAPE HANDLING.

> Regards,
> Nick Maclaren,
> University of Cambridge Computing Service,
> New Museums Site, Pembroke Street, Cambridge CB2 3QG, England.

> Tel.:  +44 1223 334761    Fax:  +44 1223 334679

 
 
 

UNIX -->EBCIDIC

Post by Nick Maclar » Wed, 07 Oct 1998 04:00:00




>WHAT AM I DOING WRONG?   ;)

>When I use ftp from Unix to AS/400 it works perfectly!
>I do not specify "ascii" although it is the default send mode.  I DO NOT
>specify binary since that precludes any translation and sends the EBCDIC
>(AS/400) file to the ASCII machine "as is".  I send files regularly both
>ways and my only complaint is the speed 250K give or take.  Of course
>"stream" data has always eaten the S/36 and AS/400 machines alive.  They
>really prefer record fixed length format...

Well, to answer your question, you are making a classic error of the
inexperienced.  You are imagining that, because one simple example
'works', there isn't a problem.

Quote:>Maybe I remembered to send files without "packed" numerics...

That would cause confusion.  So would ANSI control characters (does
the AS/400 have them? - MVS does), any characters outside the base
FTP set (and EBCDIC has several), any files written in a code page
not expected by FTP (and there are lots), any files with embedded
control characters (especially CR-LF), any files with long lines
(MVS allows over 2^40, but most Unix utilities have very low limits),
any files with structuring, and so on.

When your data matter, it isn't whether something works in the the
simple cases that is the issue.  It is whether it will either handle
or diagnose the complex cases correctly - and FTP doesn't.  Of
course, if your data DON'T matter, then what the hell?

Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QG, England.

Tel.:  +44 1223 334761    Fax:  +44 1223 334679

 
 
 

1. Wanted: <><><> Unix Specialist <><><>

PERMANENT, FULL-TIME, POSITION.

Attractive Palm Beach County, Florida Location.    MAJOR EMPLOYER.

<><><>  UNIX SPECIALIST <><><>

-- AIX, SCO Preferred
-- Programming Background Preferred
-- Knowledge of INFORMIX  a real plus

FOR CONFIDENTIAL CONSIDERATION, PLEASE CONTACT:
(resumes..in .txt format please..before calls...please)

Richard
ENTERPRISE SOFTWARE STRATEGIES, INC.

Toll-free Voice: 1-800-624-2944
Toll-free Fax:   1-800-470-3751
Voice:            1-561-998-2823
Fax:              1-561-995-0699


2. HELP: FREEZE-UP doing multiple X.25 file transfers

3. UNIX -> MSDOS -> UNIX File transfers

4. kernel BUG at arch/i386/kernel/apm.c:1699!

5. please help, problem with Unix --> NT--> Unix

6. mmap (segmentation fault) on Dell Dimension 4550

7. COMPUTERWORLD: HP from Unix -> NT -> Unix

8. Multiboot not working

9. >>PW:C/C++ UNIX ANALYST/PROGRAMMER: Unix, Sun, Solaris 2.x, C/C++

10. itoa() in C for unix >>>

11. Win95 -> telnet -> unix -> 4gl (function keys)

12. <<< Unix Support and Inventory/Purchasing £20-30k >>>

13. Win95 -> telnet -> unix -> 4gl (function keys)