Keith Wood - Data Herding: The art of EDI

Copyright 2021 by Keith Wood. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, without the prior written permission of the author.

Cover design by Klassic Designs (via the 99 Designs website)

Cover design has been designed using resources from Freepik.com

Edited by Jo Finchen-Parsons

Thanks to Jo Finchen-Parsons, via Reedsy, for looking through this and making it better. https://reedsy.com/#/freelancers/jo-f

Interior layout by Olivier Darbonville

Printed in the United States of America

CONTENTS

ABOUT THE AUTHOR

M y name is Keith Wood and I came of age a little before the internet became a thing. I feel I made the right call and got a degree in computer science in the early eighties (I was wavering between computer science and wildlife biology). I have been fortunate enough to remain employed in the technology sector since then. Im not sure that I would call myself a geek or a nerd but I will admit that others do. My wife does. I do own a metal detector (two actually) and I like to use them on the beach.

Over the past thirty-five years I have worked in different industries and different roles. I have worked in both the manufacturing and the servicing sector. I have worked in the textile, pharmaceutical, and financial services industries. I have seen a lot of things, both good and bad, but I have never been bored. I have been exchanging data between different systems for most of my career. I believe that I have made many, if not all, of the mistakes that can be made when integrating systems. In this book I will take the lessons that I have learned from those mistakes and help you avoid them.

There are unique challenges when integrating your system with an external customers system, which you may have no knowledge of. Most people who have never done this type of work cant really appreciate what has to be done. Integrating with a system which you have never seen can be difficult. Sometimes you have to try and crawl inside your trading partners head to understand what they are seeing. Some of what we do can be frustrating but some of it can be entertaining as well.

I work with technology everyday yet I am still amazed on a regular basis by things I see people accomplishing with it - both helping and hurting our world.

ABOUT THE BOOK

T his book contains all the lessons that I have learned over the years; lessons learned mainly by doing things the wrong way. In putting this together I hope to help people not repeat the same mistakes. I want this book to be relevant and useful for a long period of time, so I will stay away from making recommendations on what tools or systems you should purchase or use. Here, I am interested in sharing how to exchange data well, not how to carry out a particular task with a certain standard or on a specific platform. Advice on tools to use that is good today may not be so tomorrow and certainly will not be a few years from now.

This information should be helpful for anyone working in the EDI space; regardless if you are using a nice expensive third-party product with all the bells and whistles, or a bunch of shell scripts that you have hacked together over the years. I am trying to address challenges everyone is going to face and offer some potential solutions. More than anything I am trying to point out things you need to think about as you are interfacing with your external trading partners.

This book is not a deep dive into the complexities of the X12 standard and how purchase orders and invoices get generated and processed. Those things are beyond the scope of this book. This book will not walk you through step by step on mapping an 810 document into an XML format. There are many tools available and it would not be possible for me to cover all of the technical steps required to get each of them up and running. You will have to work through that part yourself.

Electronic Data Interchange (EDI) is what feeds data into, out of, and between systems - the plumbing of a software system, the grease that makes the wheels turn. EDI should be thought of as a utility just like the electricity in your home: you expect it to function in the background, providing what you need with minimal fuss. The greatest reward you can expect for doing a good job is to not be noticed. A lot of the time you will be working at the base metal level of a computer system. Some people like this kind of work, but there are also a lot that dont. You will probably work with people that have been assigned to your group that will not be happy about it and will be looking to get moved out as soon as possible.

I hear EDI referred to in ways that I dont agree. I hear that EDI applies to documents such as purchase orders, invoices, and other transactional documents. I have heard that EDI competes with XML, and that it competes with APIs (Application Programming Interfaces). So, in this sense I believe people are viewing EDI as the same thing as the X12 standard. Even though there are other standards that compete with X12; EDIFACT and VDA for example.

In this book, to remain true to the purpose of helping readers employ best practices and avoid the many pitfalls I have experienced over the years, I have chosen to view EDI simply as what it name states; moving data back and forth between different systems, regardless of what the data looks like as it is being interchanged.

I do not see EDI as competing with XML. XML is just a way of formatting data when you are exchanging that data electronically with a business partner.

What EDI is:

EDI will keep you engaged with your external customers.

EDI will allow you to have knowledge of most business processes within your organization.

EDI will allow you to take part with standard setting organizations, to the extent that you want.

EDI will allow you to fly under the radar (unless things go wrong).

EDI will get you blamed for a lot of things that you are not responsible for, nor that you have any control over.

EDI will allow you to be amazed by what your trading partners do at times.

What EDI is not:

EDI is not something that is going to get you exposure to the latest and greatest technology frameworks.

It is not something that you can point to and say look at what I did.

It is not something that is going to bring you a lot of awards or recognition.

Lets get started with the foundation of any good data exchange; standards. When everyone uses them, life is wonderful.

1. STANDARDS

S tandards are good. When people use standards, it is great. If people were to use standards in a standard way it would be Nirvana. Alas, Nirvana is a mythical place that doesnt really exist. There are a lot of people that put in a great deal of time and effort to develop standards to hopefully meet the needs of business. When these get turned loose in the wild, however, you will be amazed at how things actually get implemented.

I would like to give one example of something that I have personally had to deal with, where a standard was not used properly. I was implementing an interface with a major vendor in the mortgage industry. The standard that was being used was a good choice that had all of the data points defined. There just needed to be a way to transmit data about a new government program that had come out to help home owners who were underwater (owed more on their house than what it was actually worth). There was a data field already defined, implemented, and used throughout the industry to represent a mortgage insurance policy number. Instead of using that field to send the policy number, however, the vendor decided to send that critical piece of data in another field that was normally used to transmit the name of the system used to underwrite the loan. This was decided by the vendor within their development group with no discussions with any of their trading partners. Once announced there was no negotiating the decision. Making a change would have required the vendor to go back and do more development work which they were unwilling to do. Since they were such a major player in the industry there was no choice but to go back and create special logic, just for them, to map that data item. This was necessary work for all parties that were integrated with the vendor, which was everyone in the mortgage insurance industry.