My thoughts as an enterprise Java developer.

Showing posts with label original. Show all posts
Showing posts with label original. Show all posts

Saturday, March 06, 2021

Code Review Suggestions

Goal: identity significant problems so they can be fixed before deployment.

 

Ensure the code is:

  • Bug free (for bugs with a significant chance of happening)
  • Generally meets the goals of the change
  • Has appropriate automated testing
  • Doesn’t cause unexpected problems
  • Doesn’t decrease security, scalability, robustness, etc
  • Maintainable and understandable by future developers
  • Code technique generally fits with the rest of the code in the repository.
  • No unreasonable risks
  • Answers important “why” questions. Code inspection often easily answers “What” but look for important unanswered “Why” questions

Reviewers aren’t expected to:

  • Find all bugs
  • Enforce style (formatting, way of doing things, etc) as long as it fits reasonably with the rest of the codebase

Suggestions:

It is good to make optional suggestions as long as it is clear that they are optional and don’t affect approval.

 

Thursday, February 18, 2021

Command line tool Introduction

 

Goal

Introduce some command line tools and how that can work together to solve common problems.

Notes

There are many commands included with a *nix system that are useful for research. The commands may have differences on different systems so the following works on Mac OS X.

Read the links to learn more about the tools – they can do much more than is presented here.

There are more efficient ways to achieve the same results as some of these example but the examples were chosen to be a simple introduction that focuses on how the tools can be used together.

Sample files based on https://www.w3schools.com/xml/cd_catalog.xml.

 

Finding things in files

grep searches files for lines that match a pattern. It uses regular expressions.

Example: Find XML values of "EU"

grep ">EU<" *.xml

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog2.xml: <COUNTRY>EU</COUNTRY>

Linking multiple commands together

A command line pipe allows you to pass the results of one command to another command. This allows you to link many simple programs together to complex solutions in the same way that 26 letters can be put together to make millions of words.

Example: Find XML values of "EU" that also are in the N1 segment

grep ">EU<" *.xml | grep "<COUNTRY>"

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog2.xml: <COUNTRY>EU</COUNTRY>

 

Check your command by looking at the first few results

head shows only the first 10 results.

Example: Find XML values of "EU" and look at the first few results.

grep ">EU<" *.xml | head

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog.xml: <COUNTRY>EU</COUNTRY>

cd_catalog2.xml: <COUNTRY>EU</COUNTRY>

Make changes to results

sed allows you to change lines. I use the search and replace feature that uses regular expressions.

Example: Find all file names with an XML value of "EU".

grep ">EU<" *.xml | sed "s/:.*//"

cd_catalog.xml

cd_catalog.xml

cd_catalog.xml

cd_catalog.xml

cd_catalog.xml

cd_catalog2.xml

Run a command against many files.

xargs allows you to run a command against many files.

Example: Find all file names with an XML value of "EU" and find all YEAR XML nodes in those files (different lines from the EU).

grep ">EU<" *.xml | sed "s/:.*//" | xargs grep "<YEAR>" | head

cd_catalog.xml:    <YEAR>1985</YEAR>

cd_catalog.xml:    <YEAR>1988</YEAR>

cd_catalog.xml:    <YEAR>1982</YEAR>

cd_catalog.xml:    <YEAR>1990</YEAR>

cd_catalog.xml:    <YEAR>1997</YEAR>

cd_catalog.xml:    <YEAR>1998</YEAR>

cd_catalog.xml:    <YEAR>1973</YEAR>

cd_catalog.xml:    <YEAR>1990</YEAR>

cd_catalog.xml:    <YEAR>1996</YEAR>

cd_catalog.xml:    <YEAR>1987</YEAR>

Sort the results

sort allows you to sort your results.

Example: Find all file names with an XML value of "EU" and find all years in those files.

grep ">EU<" *.xml | sed "s/:.*//" | xargs grep "<YEAR>" | sed "s/:.*\">/:/;s/<\/.*//" | sort | head

cd_catalog.xml: <YEAR>1968

cd_catalog.xml: <YEAR>1968

cd_catalog.xml: <YEAR>1968

cd_catalog.xml: <YEAR>1968

cd_catalog.xml: <YEAR>1968

cd_catalog.xml: <YEAR>1971

cd_catalog.xml: <YEAR>1971

cd_catalog.xml: <YEAR>1971

cd_catalog.xml: <YEAR>1971

cd_catalog.xml: <YEAR>1971

Remove duplicates

uniq allows you to remove duplicate results (that are right next to each other). This is often used after sort.

Example: Find all file names with an XML value of "EU" and find all unique sorted years in those files.

grep ">EU<" *.xml | sed "s/:.*//" | xargs grep "<YEAR>" | sed "s/:.*\">/:/;s/<\/.*//" | sort | uniq | head

cd_catalog.xml: <YEAR>1968

cd_catalog.xml: <YEAR>1971

cd_catalog.xml: <YEAR>1973

cd_catalog.xml: <YEAR>1982

cd_catalog.xml: <YEAR>1983

cd_catalog.xml: <YEAR>1985

cd_catalog.xml: <YEAR>1987

cd_catalog.xml: <YEAR>1988

cd_catalog.xml: <YEAR>1990

cd_catalog.xml: <YEAR>1991

Count duplicates

uniq also allows counting duplicates

Example: Find all file names with an XML value of "EU" and count years per file.

grep ">EU<" *.xml | sed "s/:.*//" | xargs grep "<YEAR>" | sed "s/:.*\">/:/;s/<\/.*//" | sort | uniq | sed "s/:.*//" | uniq -c15 cd_catalog.xml

3 cd_catalog2.xml

Copy intermediate results to a file

tee allows you to store the results of a command while still copying the results to the next command

Example: Find all file names with an XML value of "EU", count unique years per file, and store in yearCounts.txt.

grep ">EU<" *.xml | sed "s/:.*//" | xargs grep "<YEAR>" | sed "s/:.*\">/:/;s/<\/.*//" | sort | uniq | sed "s/:.*//" | uniq -c | tee yearCounts.txt15 cd_catalog.xml

3 cd_catalog2.xml

Count the number of results

wc allows you to count the results.

Example: Find all file names with an XML value of "EU", count how many have exactly 3 unique years.

grep ">EU<" *.xml | sed "s/:.*//" | xargs grep "<YEAR>" | sed "s/:.*\">/:/;s/<\/.*//" | sort | uniq | sed "s/:.*//" | uniq -c | tee yearCounts.txt | grep "^ *3 " | sed "s/^ *3 //;s/\.xml$//" | tee 3years.txt | wc -l

1

Educational Pivot Points

Looking back at what brought me to where I am, there are many events that were important.

My older brother had trouble learning to read and I got the same 1st grade teacher as he had. She assumed I also would have trouble learning to read and refused to teach me. The special education reading teacher tested me and found that I didn't need to be in a special class. Since there was no place for me to learn to read, the special education reading teacher taught me individually. That individual instruction was better than the normal options and by the 3rd grade I was tested to be reading at the 10th grade level. My 1st grade teacher's negative attitude looked like it would hurt me, but someone else helped me do better then she could have likely done!

 

In 3rd grade, I remember being taught about rounding numbers. While it was being explained, I wondered what to do about the number 5. I was told to always round it up but in college (chemistry or physics), I was told to round 5 to get to an even number so that the rounding errors for 5 are canceled out. Over time you will find that it becomes more complex for figure out the right thing to do.

 

In 4th grade, I became friends with a boy who was interested in computers and did a little programming. I found that I also liked computers and programming. Seemingly accidental events can lead you on interesting paths.

 

In 6th grade, my teacher would sometimes play football with us during recess. Even though I never was a great athlete, he still occasionally threw the ball to me. I really appreciated that and you never know when a small act of kindness is meaningful.

 

At the start of 7th grade, we moved to a new state. I had been in an advanced math class but the new school wouldn't do that so I was bored with many of my classes that year. Sometimes you need to deal with setbacks.

 

In 7th or 8th grade, I started volunteering in the computer lab and was unofficially allowed to bring a computer home for the summer and got a summer job cleaning computers.

 

In 8th grade, they let me take 9th grade math. If you deal with setbacks well, that may open up doors. In that math class, I met a friend that I still talk with occasionally. He also introduced me to Boy Scouts which I enjoyed doing for many years and helped me become a lifeguard.

 

In 9th grade they upgraded the computer lab and installed the first computer network at that school. I volunteered to help install that and learned a lot. The computer coordinator even took me out of classes so I could learn when someone was brought in to teach him. Once that trainer told me to copy a file and I had to ask how to do that. Everyone starts with 0 knowledge so don't be embarrassed to ask for something to be explained. That new lab and network wasn't used for a semester so I got to explore it and help figure out how to use it. Because I had much more free time, I soon knew much more about it than the computer coordinator and he allowed me to have full administrative privileges. This included the ability to see and change any grades that teachers stored on the network but I never exploited that. With great power comes great responsibility. How you use power shows a lot about your character. I also learned a lot because I always stayed 2 hours after school in the computer lab working on homework, learning, and helping other students. I had to learn how to meekly help train my teachers and help them with any of their problems. I know some of the teachers and school administrators didn't like that I had that much power so I had to meekly show them that I was only doing good. Sometimes in classes in the computer lab, I had to lead part of the class or tell the teacher what to do. Some of my class homework was done with computers when none of the other students used computers. I suspect that my work somewhat dazzled the teachers and got better grades that other students who had better content. Don't be dazzled by flashy presentation that lacks good content. I also took 10th and 11th grade math, 10th grade science, and a German class usually taken by 10-12th graders so I had to learn to be humble to not make the students in higher grades feel bad. In my biology class, I was at a table with a student who didn't do well in that class. He used to gleek on me but I knew he must feel bad about looking bad compared to someone in a lower grade.

 

In 10th grade, my math class was for seniors and even had my older brother in it. That class was quite a bit different than my other math classes and at first I didn't do great so I had to put in a lot more work and studying than I used to.

 

In 11th grade, I was done with all of their math classes so they gave me a calculus textbook and put me in an Algebra 2 class (that I had taken 2 years ago). It was really hard to concentrate when the teacher was lecturing so I made very slow progress that year. Because of my good work with computers, the computer coordinator went to bat for me against the school board who didn't like my access.

 

For 12th grade, we asked the school board to pay for me to take a math, science, and computer science class at a nearby university but they only approved the computer science class. The district already had a policy allowing seniors to skip the last semester if they had completed enough work so I started college full time my last senior semester. When I took college calculus 1, my previous self study helped me for a few weeks but I had to work harder after that. When I started college, I looked at their computer network and found that it was the same type as I had used in high school (but more complex). While looking around, I found 1 department didn't have a password on the account with full privileges so I sent them a message warning them about that. They replied with a job offer! The department was the same department with my major so it was very convenient to mix working with my school work. While I was good at English, I wasn't interested in advancing that area so I signed up for the easiest English class required. After the first week, my teacher told me that I could take the honors English class and that it would involve less writing so I switched to that class.

 

When I was a sophomore, Mayo Clinic called the department looking for a summer intern. Because of my work in the department, they recommended me. That turned into summer and part time work the rest of my college career and a job offer after. During the first few weeks I had to speak up in a meeting to tell others I barely knew that they were wrong about a technology decision. You have to be meek and knowledgeable to make that work!

 

As a junior while working at Mayo I started using a new programming language that is still my primary language. Even though I never took any classes on that programming language, my first job was in that language. The class that was the most useful to my job programming was actually a math class with no programming -- it was about proofs and I've found that the proof process has been very useful when I program. One class had a semester project and I decided to use the new programming language. Because I liked programming and the new language had many features, I made quick progress. A few weeks into the class, the teacher gave help in the old language that I didn't use for problems that I had already addressed! Working on interesting problems can cause you to spend a lot of time on them and make quick progress.

 

It took me 4.5 years to graduate from college but I went a little slower because of working a lot at Mayo. That slowdown actually allowed me to start looking for a job with 2 years of experience. Since I lived at home during college, had the side jobs, and was frugal, I was able to graduate with no debt. When I was looking for a job, there were 3 related job types that I considered. One of those types doesn't exist anymore so I'm glad I didn't choose that.

 

When I interviewed at SPS Commerce, I interviewed for 2 different departments. After some thought, I picked one and the other was sold to a different company a few years later. After a few months, my manager gave me a problem that we didn't know how to reproduce, only happened in the production environment, only happened sometimes, and then left for a week to go to a conference. I had no idea how to fix it but I knew a step that I could take that might help. After a few times of just taking the next step, I finally was able to find the cause of the problem and figure out a fix. You don't need to know how to solve all problems if you just know the next step and continue taking each next step. A few months after starting, my manager asked if I would like to be moved to a new responsibility (production support). I wasn't sure if I would like that but I decided to give it a try. A few years later the company went through layoffs but the production support developer is the last person to be laid off so I was spared from all layoffs. After a year, I was promoted.

 

Early on I led a small team. Because I have a long-term focus, and don't want to get the same problem twice, I was able to eventually handle production without a team while handling new features. I have worked on small teams, and led small teams, but normally work alone. Because of this, I handle everything from architecture, design, coding, debugging, and research to working with quality assurance, customer support, and product management. I long ago decided that I didn't want to go into management because I saw that they usually couldn't spend much time solving technical problems and I really enjoy that. I like working on more complex and harder problems and features.

 

A decade ago, I started working on my master's degree. It took me a while to find a program that I thought would be applicable to my job and teach me new things. I stopped after taking only 2 classes because I couldn't find more useful classes.

 

Of my 22 years at SPS Commerce, 20 of them have been working on the same service but no one else has worked on that service for more than a few years. That level of experience allows me to know the service very well, quickly give very authoritative answers to questions about the service, see the service go through huge changes, make and execute long term plans for the service, quickly find problems, quickly make changes, help other developers efficiently work with the service, and advise others about their work in relation to the service (e.g. I often work with product management to improve their suggested changes). I am now an extreme outlier in my industry where most people only stay 3-5 years at a job.

 

When I started, most of my time was spent coding features or fixes that others gave to me but now I code 5% of my time. I now spend my time researching and planning changes, training others, and researching complex problems/questions others have about the service. In real life it is almost impossible to make something 1,000 times better, but I can occasionally make something literally 1,000 better (e.g. take a process that takes hours for customers and make it take seconds). Most of my time at SPS, I have telecommuted 1-3 days/week so I have actually enjoyed the move to 100% telecommuting during COVID-19. For me, the commuting time savings is equivalent to an extra 8 weeks of vacation annually! I live in a sparsely populated area so staying away from people isn't very constraining.

 

I would say that I'm almost addicted to learning. After formal education ended, I still sought out information on improving my skills but that information gets harder to find as my level of knowledge deepens. Looking back, the most useful parts of my education often weren't obvious until much later so it would be hard to know how to go back and be sure that a change would improve that. The things I liked best where when I could explore a new area -- from first learning programming to learning the computer network at school to learning a new service at SPS Commerce to learning new ways to solve problems.

 

Friday, January 22, 2021

Naming variables

It isn’t very useful to name variables based on information that can be easily determined on the declaration line. It is useful to name variables based on the purpose of the variable. Otherwise the purpose may be spread out across the usage of the variable.

In code it is often easy to answer “what” but hard to answer “why” so the code should be written to answer “why”.

Wednesday, May 20, 2020

Logging

Level When to use Example Alert Response Environments
Fatal The system can't run Can't connect to main DB Immediately Immediately address All
Error There is a system problem Unable to run a DB query If the count passes a small threshold Keep current on all errors with plan for addressing and escalate as appropriate All
Warning There is a problem that isn't a system problem A specific request took longer than expected to process If the count passes a big threshold Generally stay current with most common logs and optionally create plan and/or escalate as appropriate All
Info Something happened that isn't a problem but is noteworthy Unable to process request because the request was invalid Optionally if the count passes a huge threshold Occasionally review to look for patterns that may need to be addressed Probably prod. Usually test. Always Dev.
Debug Something not noteworthy but useful for inspecting how the system is running A specific query took x milliseconds to run Never None Prod only rarely for some loggers. test rarely. Dev often.

Friday, December 23, 2016

Better show that a password is being entered

Terminals need a better way to show that a password is being enter in order to reduce the chance that a password would be accidentally entered somewhere incorrect.
The whole screen should have some indication that a password is being entered. I.e. Dim everything but the password field. This would reduce the problem of accidentally typing your password into chat when you thought another window was active.

To log or not to log

To log or not to log?
That is the question.
Whether is nobler to in the Sumo to suffer ups and downs of outrageous quantity, or to take out logs against a Sea of repetition.

Wednesday, August 10, 2016

Tight scoping of constants

How do you declare constants that are only needed in a small scope (function or smaller)? I declare them in the tightest scope as final with an all-caps name. I.e.:

final int THRESHOLD = 1000;

That coveys the intent that it is a constant but minimizes scope.

Disable new UI components until the user has had time to react to them

How often do you use a program, decide to take an action (I.e. Press Return), and then have something popup and use your action before you even have a chance to read the popup? I hate when that happens because I don’t even know what I told the computer to do.

Input on a new UI element should be disabled for a short time to ensure that the user actually wants to take action on the new UI element. The delay has to be just long enough for the user to realize that the action may do something differently and not too long to slow down users who know that the popup is coming. So the delay should probably be a few hundred milliseconds.

Tuesday, January 14, 2014

Preferences/Settings should be minimized

Allowing a user to change how an application works by only going into preferences/settings should be avoided.

  • Most users won't even look there so features will be underused which is a waste of development resources.
  • Try to prompt the user to change their preference when they do a related action. i.e. if there is a preference for the number of items to show on a page and the user changes the dropdown to show a different number of items on the page then ask the user if they would like to update the preference to the new value.
  • When prompting the user, avoid making the user click an extra time. Instead of doing a popup, just add text that prompts the user.

Tuesday, December 10, 2013

More interesting wait screen

I have seen many wait screens that do the job but are boring. I think it might work a lot better to make a wait screen that displays something interesting. Maybe draw a fractal or present a simple game. If there is something interesting, the user shouldn't mind the wait as much.

Thursday, October 17, 2013

The type of work that I like to do

I like working on detailed, complex, and/or interesting problems on a product that I know well, has a large codebase, and has to support high load. I call those “guru-level” problems.

Tuesday, October 15, 2013

Change comments to logs?

It seems that almost all of the time, comments in a method work better as logging statements.
Reasons:

  • The log statements still give clues to the developer about what is happening
  • The log statements also give that information to someone looking at the logs
  • Since log statements will be seen more, they are more likely to to kept current

Wednesday, October 02, 2013

Keeping sensitive data out of logs

When a product has logging there is a risk that sensitive data(i.e. passwords) will make it into the logs. How do we reduce that risk?

Logging an object or adding toString to a class might not obviously leak sensitive data so it is probably better to make sensitive data obvious. i.e. If sensitive data is stored in a Properties object, as soon as the properties object is obtained, it should move sensitive data to a separate location (i.e. a separate String variable in the class) and remove the sensitive data from properties so it is obvious that there is sensitive data.

Friday, September 20, 2013

Comments for people considering software engineering

I enjoy all the new problems & industry changes, that we can make a huge difference, and that it is usually quick to see the results of my work.
I didn’t expect my college degree to teach me so little of what I use each day – they are only able to teach the basics and we have to continuously learn new stuff.
I suggest going the extra mile if you only do the minimal coursework you won’t excel.  Do more and better than is required and/or side projects to push the boundaries of what you know and practice learning on your own.  Get involved in projects, research ideas, and try out new skills.

Increasing the max heap size can increase the chance of an OutOfMemoryError in Java

4.5 years ago I encountered a problem where I fixed an OutOfMemoryError by decreasing the max heap size.

Basically there is a C++ heap and if the Java heap takes up too much space then the C++ heap can get an OutOfMemoryError.  This generally happens when the java heap size is too near the maximum that that O/S allows – for Windows XP that max is in the 1.2 to 1.4 GB range so if your max heap size is 1 to 1.2 GB you may experience this problem.

Thursday, September 12, 2013

The cost of patches

Developers are willing to put almost any issue into a patch, but we want to make sure that the customer knows what a patch costs them. A patch is an extra build that causes development hours of extra time to create, Q.A. hours of extra time to test, and extra time to install. We must always balance having some issues into production faster with having more issues get done in the long run.

Tuesday, July 23, 2013

Registry Hacks

These are some of my favorite Windows registry additions:

Add Notepad as a right click menu option to open any file:
CLASSES_ROOT\*\shell\notepad\command\:REG_EXPAND_SZ: %SystemRoot%\system32\notepad.exe %1

Add DosHere as a right click menu option to all folders to start a command prompt in that directory:
CLASSES_ROOT\Directory\shell\DosHere\command\:REG_EXPAND_SZ: %SystemRoot%\system32\cmd.exe /k cd "%1" && title %1

Tuesday, June 11, 2013

Deprecation Plan


Once a method is deprecated, Java seems to have no further plans for removing it before Java 2.0 (if that ever happens).

I suggest the following steps be done by default:

  1. Deprecate the method (Java already does this) and generate info log about using the method
  2. At the next major release, increase the logging level to warning
  3. At the next major release, increase the logging level to severe and remove from compile time libraries
  4. After 2 major releases, throw an exception in the runtime (with useful message)
  5. At the next major release, remove from the runtime

For example, the old Java 1.0 AWT event model was deprecated in 1.1. It this plan was followed, it could not have been used in code compiled in 1.3 (3 years after deprecation) and would have thrown an exception with the 1.5 runtime (7.5 years after deprecation).

Of course the timing of those steps may have to be tweaked (i.e. quick releases, widely used methods, etc) but it can be the default blueprint for getting rid of old code.

Do you prefer the current plan of leaving deprecated methods until Java 2.0 or do you think we should have the above plan or another plan for removing them.

Thursday, April 25, 2013

Interfaces: balancing debugging vs. customization

What are the cost vs. benefits of creating interfaces and how to balance them. I am only considering cases where interfaces are optional and aren’t needed.
    Benefits:
  1. Simplify interaction: Easier to see how to use a List than an ArrayList
  2. Make testing easier because replacement implementations can be used
  3. Can make future changes easier
    Costs:
  1. Harder to understand what the code is doing
  2. When you need to look at the implementing class, it can take a lot of work to find it.
According to SOLID, "one should “Depend upon Abstractions. Do not depend upon concretions.” "
The citation says: “, as much as is feasible, the principle should be followed. The reason is simple, concrete things change alot, abstract things change much less frequently. Morevoer, abstractions are “hinge points”, they represent the places where the design can bend or be extended, without themselves being modified (OCP).”

I don’t see that all or even most concrete things change a lot (at least in their public interface). If the public interface needs to change then doesn’t that mean that the interface class probably would need to change also?

How would that work with a Swing program? Would all GUI elements need to be passed in as interfaces?