My thoughts as an enterprise Java developer.

Thursday, February 18, 2021

Command line tool Introduction

 

Goal

Introduce some command line tools and how that can work together to solve common problems.

Notes

There are many commands included with a *nix system that are useful for research. The commands may have differences on different systems so the following works on Mac OS X.

Read the links to learn more about the tools – they can do much more than is presented here.

There are more efficient ways to achieve the same results as some of these example but the examples were chosen to be a simple introduction that focuses on how the tools can be used together.

Sample files based on https://www.w3schools.com/xml/cd_catalog.xml.

 

Finding things in files

grep searches files for lines that match a pattern. It uses regular expressions.

Example: Find XML values of "EU"

grep ">EU<" *.xml

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog2.xml: <COUNTRY>EU</COUNTRY>

Linking multiple commands together

A command line pipe allows you to pass the results of one command to another command. This allows you to link many simple programs together to complex solutions in the same way that 26 letters can be put together to make millions of words.

Example: Find XML values of "EU" that also are in the N1 segment

grep ">EU<" *.xml | grep "<COUNTRY>"

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog2.xml: <COUNTRY>EU</COUNTRY>

 

Check your command by looking at the first few results

head shows only the first 10 results.

Example: Find XML values of "EU" and look at the first few results.

grep ">EU<" *.xml | head

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog2.xml: <COUNTRY>EU</COUNTRY>

Make changes to results

sed allows you to change lines. I use the search and replace feature that uses regular expressions.

Example: Find all file names with an XML value of "EU".

grep ">EU<" *.xml | sed "s/:.*//"

cd_catalog.xml

cd_catalog.xml

cd_catalog.xml

cd_catalog.xml

cd_catalog.xml

cd_catalog2.xml

Run a command against many files.

xargs allows you to run a command against many files.

Example: Find all file names with an XML value of "EU" and find all YEAR XML nodes in those files (different lines from the EU).

grep ">EU<" *.xml | sed "s/:.*//" | xargs grep "<YEAR>" | head

cd_catalog.xml:    <YEAR>1985</YEAR>

cd_catalog.xml:    <YEAR>1988</YEAR>

cd_catalog.xml:    <YEAR>1982</YEAR>

cd_catalog.xml:    <YEAR>1990</YEAR>

cd_catalog.xml:    <YEAR>1997</YEAR>

cd_catalog.xml:    <YEAR>1998</YEAR>

cd_catalog.xml:    <YEAR>1973</YEAR>

cd_catalog.xml:    <YEAR>1990</YEAR>

cd_catalog.xml:    <YEAR>1996</YEAR>

cd_catalog.xml:    <YEAR>1987</YEAR>

Sort the results

sort allows you to sort your results.

Example: Find all file names with an XML value of "EU" and find all years in those files.

grep ">EU<" *.xml | sed "s/:.*//" | xargs grep "<YEAR>" | sed "s/:.*\">/:/;s/<\/.*//" | sort | head

cd_catalog.xml: <YEAR>1968

cd_catalog.xml: <YEAR>1968

cd_catalog.xml: <YEAR>1968

cd_catalog.xml: <YEAR>1968

cd_catalog.xml: <YEAR>1968

cd_catalog.xml: <YEAR>1971

cd_catalog.xml: <YEAR>1971

cd_catalog.xml: <YEAR>1971

cd_catalog.xml: <YEAR>1971

cd_catalog.xml: <YEAR>1971

Remove duplicates

uniq allows you to remove duplicate results (that are right next to each other). This is often used after sort.

Example: Find all file names with an XML value of "EU" and find all unique sorted years in those files.

grep ">EU<" *.xml | sed "s/:.*//" | xargs grep "<YEAR>" | sed "s/:.*\">/:/;s/<\/.*//" | sort | uniq | head

cd_catalog.xml: <YEAR>1968

cd_catalog.xml: <YEAR>1971

cd_catalog.xml: <YEAR>1973

cd_catalog.xml: <YEAR>1982

cd_catalog.xml: <YEAR>1983

cd_catalog.xml: <YEAR>1985

cd_catalog.xml: <YEAR>1987

cd_catalog.xml: <YEAR>1988

cd_catalog.xml: <YEAR>1990

cd_catalog.xml: <YEAR>1991

Count duplicates

uniq also allows counting duplicates

Example: Find all file names with an XML value of "EU" and count years per file.

grep ">EU<" *.xml | sed "s/:.*//" | xargs grep "<YEAR>" | sed "s/:.*\">/:/;s/<\/.*//" | sort | uniq | sed "s/:.*//" | uniq -c15 cd_catalog.xml

3 cd_catalog2.xml

Copy intermediate results to a file

tee allows you to store the results of a command while still copying the results to the next command

Example: Find all file names with an XML value of "EU", count unique years per file, and store in yearCounts.txt.

grep ">EU<" *.xml | sed "s/:.*//" | xargs grep "<YEAR>" | sed "s/:.*\">/:/;s/<\/.*//" | sort | uniq | sed "s/:.*//" | uniq -c | tee yearCounts.txt15 cd_catalog.xml

3 cd_catalog2.xml

Count the number of results

wc allows you to count the results.

Example: Find all file names with an XML value of "EU", count how many have exactly 3 unique years.

grep ">EU<" *.xml | sed "s/:.*//" | xargs grep "<YEAR>" | sed "s/:.*\">/:/;s/<\/.*//" | sort | uniq | sed "s/:.*//" | uniq -c | tee yearCounts.txt | grep "^ *3 " | sed "s/^ *3 //;s/\.xml$//" | tee 3years.txt | wc -l

1

Educational Pivot Points

Looking back at what brought me to where I am, there are many events that were important.

My older brother had trouble learning to read and I got the same 1st grade teacher as he had. She assumed I also would have trouble learning to read and refused to teach me. The special education reading teacher tested me and found that I didn't need to be in a special class. Since there was no place for me to learn to read, the special education reading teacher taught me individually. That individual instruction was better than the normal options and by the 3rd grade I was tested to be reading at the 10th grade level. My 1st grade teacher's negative attitude looked like it would hurt me, but someone else helped me do better then she could have likely done!

 

In 3rd grade, I remember being taught about rounding numbers. While it was being explained, I wondered what to do about the number 5. I was told to always round it up but in college (chemistry or physics), I was told to round 5 to get to an even number so that the rounding errors for 5 are canceled out. Over time you will find that it becomes more complex for figure out the right thing to do.

 

In 4th grade, I became friends with a boy who was interested in computers and did a little programming. I found that I also liked computers and programming. Seemingly accidental events can lead you on interesting paths.

 

In 6th grade, my teacher would sometimes play football with us during recess. Even though I never was a great athlete, he still occasionally threw the ball to me. I really appreciated that and you never know when a small act of kindness is meaningful.

 

At the start of 7th grade, we moved to a new state. I had been in an advanced math class but the new school wouldn't do that so I was bored with many of my classes that year. Sometimes you need to deal with setbacks.

 

In 7th or 8th grade, I started volunteering in the computer lab and was unofficially allowed to bring a computer home for the summer and got a summer job cleaning computers.

 

In 8th grade, they let me take 9th grade math. If you deal with setbacks well, that may open up doors. In that math class, I met a friend that I still talk with occasionally. He also introduced me to Boy Scouts which I enjoyed doing for many years and helped me become a lifeguard.

 

In 9th grade they upgraded the computer lab and installed the first computer network at that school. I volunteered to help install that and learned a lot. The computer coordinator even took me out of classes so I could learn when someone was brought in to teach him. Once that trainer told me to copy a file and I had to ask how to do that. Everyone starts with 0 knowledge so don't be embarrassed to ask for something to be explained. That new lab and network wasn't used for a semester so I got to explore it and help figure out how to use it. Because I had much more free time, I soon knew much more about it than the computer coordinator and he allowed me to have full administrative privileges. This included the ability to see and change any grades that teachers stored on the network but I never exploited that. With great power comes great responsibility. How you use power shows a lot about your character. I also learned a lot because I always stayed 2 hours after school in the computer lab working on homework, learning, and helping other students. I had to learn how to meekly help train my teachers and help them with any of their problems. I know some of the teachers and school administrators didn't like that I had that much power so I had to meekly show them that I was only doing good. Sometimes in classes in the computer lab, I had to lead part of the class or tell the teacher what to do. Some of my class homework was done with computers when none of the other students used computers. I suspect that my work somewhat dazzled the teachers and got better grades that other students who had better content. Don't be dazzled by flashy presentation that lacks good content. I also took 10th and 11th grade math, 10th grade science, and a German class usually taken by 10-12th graders so I had to learn to be humble to not make the students in higher grades feel bad. In my biology class, I was at a table with a student who didn't do well in that class. He used to gleek on me but I knew he must feel bad about looking bad compared to someone in a lower grade.

 

In 10th grade, my math class was for seniors and even had my older brother in it. That class was quite a bit different than my other math classes and at first I didn't do great so I had to put in a lot more work and studying than I used to.

 

In 11th grade, I was done with all of their math classes so they gave me a calculus textbook and put me in an Algebra 2 class (that I had taken 2 years ago). It was really hard to concentrate when the teacher was lecturing so I made very slow progress that year. Because of my good work with computers, the computer coordinator went to bat for me against the school board who didn't like my access.

 

For 12th grade, we asked the school board to pay for me to take a math, science, and computer science class at a nearby university but they only approved the computer science class. The district already had a policy allowing seniors to skip the last semester if they had completed enough work so I started college full time my last senior semester. When I took college calculus 1, my previous self study helped me for a few weeks but I had to work harder after that. When I started college, I looked at their computer network and found that it was the same type as I had used in high school (but more complex). While looking around, I found 1 department didn't have a password on the account with full privileges so I sent them a message warning them about that. They replied with a job offer! The department was the same department with my major so it was very convenient to mix working with my school work. While I was good at English, I wasn't interested in advancing that area so I signed up for the easiest English class required. After the first week, my teacher told me that I could take the honors English class and that it would involve less writing so I switched to that class.

 

When I was a sophomore, Mayo Clinic called the department looking for a summer intern. Because of my work in the department, they recommended me. That turned into summer and part time work the rest of my college career and a job offer after. During the first few weeks I had to speak up in a meeting to tell others I barely knew that they were wrong about a technology decision. You have to be meek and knowledgeable to make that work!

 

As a junior while working at Mayo I started using a new programming language that is still my primary language. Even though I never took any classes on that programming language, my first job was in that language. The class that was the most useful to my job programming was actually a math class with no programming -- it was about proofs and I've found that the proof process has been very useful when I program. One class had a semester project and I decided to use the new programming language. Because I liked programming and the new language had many features, I made quick progress. A few weeks into the class, the teacher gave help in the old language that I didn't use for problems that I had already addressed! Working on interesting problems can cause you to spend a lot of time on them and make quick progress.

 

It took me 4.5 years to graduate from college but I went a little slower because of working a lot at Mayo. That slowdown actually allowed me to start looking for a job with 2 years of experience. Since I lived at home during college, had the side jobs, and was frugal, I was able to graduate with no debt. When I was looking for a job, there were 3 related job types that I considered. One of those types doesn't exist anymore so I'm glad I didn't choose that.

 

When I interviewed at SPS Commerce, I interviewed for 2 different departments. After some thought, I picked one and the other was sold to a different company a few years later. After a few months, my manager gave me a problem that we didn't know how to reproduce, only happened in the production environment, only happened sometimes, and then left for a week to go to a conference. I had no idea how to fix it but I knew a step that I could take that might help. After a few times of just taking the next step, I finally was able to find the cause of the problem and figure out a fix. You don't need to know how to solve all problems if you just know the next step and continue taking each next step. A few months after starting, my manager asked if I would like to be moved to a new responsibility (production support). I wasn't sure if I would like that but I decided to give it a try. A few years later the company went through layoffs but the production support developer is the last person to be laid off so I was spared from all layoffs. After a year, I was promoted.

 

Early on I led a small team. Because I have a long-term focus, and don't want to get the same problem twice, I was able to eventually handle production without a team while handling new features. I have worked on small teams, and led small teams, but normally work alone. Because of this, I handle everything from architecture, design, coding, debugging, and research to working with quality assurance, customer support, and product management. I long ago decided that I didn't want to go into management because I saw that they usually couldn't spend much time solving technical problems and I really enjoy that. I like working on more complex and harder problems and features.

 

A decade ago, I started working on my master's degree. It took me a while to find a program that I thought would be applicable to my job and teach me new things. I stopped after taking only 2 classes because I couldn't find more useful classes.

 

Of my 22 years at SPS Commerce, 20 of them have been working on the same service but no one else has worked on that service for more than a few years. That level of experience allows me to know the service very well, quickly give very authoritative answers to questions about the service, see the service go through huge changes, make and execute long term plans for the service, quickly find problems, quickly make changes, help other developers efficiently work with the service, and advise others about their work in relation to the service (e.g. I often work with product management to improve their suggested changes). I am now an extreme outlier in my industry where most people only stay 3-5 years at a job.

 

When I started, most of my time was spent coding features or fixes that others gave to me but now I code 5% of my time. I now spend my time researching and planning changes, training others, and researching complex problems/questions others have about the service. In real life it is almost impossible to make something 1,000 times better, but I can occasionally make something literally 1,000 better (e.g. take a process that takes hours for customers and make it take seconds). Most of my time at SPS, I have telecommuted 1-3 days/week so I have actually enjoyed the move to 100% telecommuting during COVID-19. For me, the commuting time savings is equivalent to an extra 8 weeks of vacation annually! I live in a sparsely populated area so staying away from people isn't very constraining.

 

I would say that I'm almost addicted to learning. After formal education ended, I still sought out information on improving my skills but that information gets harder to find as my level of knowledge deepens. Looking back, the most useful parts of my education often weren't obvious until much later so it would be hard to know how to go back and be sure that a change would improve that. The things I liked best where when I could explore a new area -- from first learning programming to learning the computer network at school to learning a new service at SPS Commerce to learning new ways to solve problems.