Java Developer Central

Apache Commons IOUtils

Introduction

The Apache Commons includes Apache Commons IO. It is a library of utilities to help with various IO functionalities. There are a lot of utility classes and IOUtils is one of them. In this post, we will see about the Apache Commons IO’s IOUtils class and the static utility methods in it.

Maven and Gradle dependency on Apache Common IO

You can import Apache Commons IO from Maven as shown below.

<!-- https://mvnrepository.com/artifact/commons-io/commons-io -->
<dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>2.7</version>
</dependency>

You can find the latest version of commons-io from Maven.

For gradle,

compile 'org.apache.commons:commons-io:2.7'

Apache Commons IO IOUtils methods

The rest of this post explores the utility methods in the IOUtils class. At a high level, we will see the methods that enable us in reading, writing, and copying data. Also, we will see other helpful methods to buffer, to convert data from one form to another (like InputStream to byte array).

For this post, I’ll be mostly using a file called sample-file.txt – assume it has the below data in it. It represents the id of the student, the student name and the overall percentage.

1-Will-90
2-James-87
3-Kyle-67

InputStream/Reader and OutputStream/Writer

All the methods in the IOUtils operate on an InputStream or a Reader and an OutputStream or a Writer. Thus, the utility methods will appear symmetric i.e., whatever operations they support on an InputStream will be available on a Reader and features provided on an OutputStream will be similarly provided for a Writer instance as well.
Also, all of the methods which we will see will throw an IOException. I’ll not show it to reduce the verbosity.

IOUtils Buffer

The IOUtils.buffer method creates a Buffered Stream/Buffered Reader/Buffered Writer. 

  • Takes an InputStream and returns a BufferedInputStream.
  • Takes a Reader and returns a BufferedReader.
  • Accepts an OutputStream and returns a Buffered OutputStream.
  • Accepts an Writer and returns a BufferedWriter.
Let us look at examples for these.
//InputStream
InputStream inputStream = new FileInputStream(".../data/sample-file.txt");
BufferedInputStream bufferedInputStream = IOUtils.buffer(inputStream);
 
 //We can test the contents by printing it as a String as shown below
System.out.println(IOUtils.toString(bufferedInputStream, StandardCharsets.UTF_8));

//Reader
Reader reader = new FileReader(".../data/sample-file.txt");
BufferedReader bufferedReader = IOUtils.buffer(reader);


//OutputStream
OutputStream outputStream = new FileOutputStream(".../data/sample-output-file.txt");
BufferedOutputStream bufferedOutputStream = IOUtils.buffer(outputStream);
//Can write into the output stream
bufferedWriter.write('a');
bufferedWriter.close();

//Writer
Writer writer = new FileWriter(".../data/sample-output-file.txt");
BufferedWriter bufferedWriter = IOUtils.buffer(writer);

Other variations

There are overloaded buffer methods that take a size parameter. It is the buffer size of the created BufferedInputStream or BufferedOutputStream or BufferedReader/ BufferedWriter.

IOUtils.buffer(inputStream, 10);
IOUtils.buffer(reader, 10);
IOUtils.buffer(outputStream, 10);
IOUtils.buffer(writer, 10);

Reading data using IOUtils

Reading from an InputStream and a Reader

There are methods to read data from an InputStream or a Reader into a byte array or a character array respectively. It will try to read as many bytes/characters as possible. It returns the number of bytes/characters read. We will use the data set mentioned earlier.

InputStream inputStream = new FileInputStream(".../data/sample-file.txt");
byte[] bytes = new byte[7]; //Creating a byte array of size 7.
int numberOfBytesRead = IOUtils.read(inputStream, bytes);
System.out.println(numberOfBytesRead);//7
System.out.println(new String(bytes)); //1-Will-

It will read 7 bytes because that is the size of the byte array passed to the read() method. The read bytes are printed above.

Similarly, we can use a Reader as shown below.
Reader reader = new FileReader(".../data/sample-file.txt");
char[] chars = new char[7];
int numberOfCharsRead = IOUtils.read(reader, chars);
System.out.println(numberOfCharsRead);//7
System.out.println(new String(chars)); //1-Will-

Using offsets and lengths

We can pass an offset and a length parameter to the read method. The offset is the offset into the buffer and the length denotes the number of bytes/characters to read.

InputStream inputStream = new FileInputStream(".../data/sample-file.txt");
bytes = new byte[7];
numberOfBytesRead = IOUtils.read(inputStream, bytes, 1, 5);
System.out.println(numberOfBytesRead);//5
System.out.println(new String(bytes)); //1-Wil

We are passing a new byte[] (uninitialized) with offset as 1 and length as 5. Hence, it will start populating the byte array from index 1 and writes 5 bytes. The first and the last index of the byte[] will by 0 (the default value for a byte if it is not initialised). That when converted to a String will be the character `\u0000`. 

Note: \u0000 is printed as a little box in Intellij IDE. It may not be printed depending on your console.
 
Similarly, for a reader,
 
Reader reader = new FileReader(".../data/sample-file.txt");
chars = new char[7];
numberOfCharsRead = IOUtils.read(reader, chars, 0, 5);
System.out.println(numberOfCharsRead);//5
System.out.println(new String(chars)); //1-Wil

Here, we start at index 0 and read 5 characters, leaving the last two bytes uninitialized to 0.

Read Fully

This is similar to the read method, but it attempts to read fully. In other words, it fails with an EOFException if there aren’t enough bytes or characters to remaining in the source (input stream or reader) to fill the byte or the character array.

InputStream inputStream = new FileInputStream(".../data/sample-file.txt");
byte[] bytes = new byte[7];
IOUtils.readFully(inputStream, bytes);
System.out.println(new String(bytes)); //1-Will-
//Above is similar to the read method 

InputStream inputStream = new FileInputStream(".../data/sample-file.txt");
bytes = new byte[70];
IOUtils.readFully(inputStream, bytes); //java.io.EOFException: Length to read: 70 actual: 31

The second part of the above snippet creates a byte array that is larger than the file content. Hence, it cannot be filled up (read fully). Thus, the readFully call (last line) will throw an EOFException.

For a reader,

Reader reader = new FileReader(".../data/sample-file.txt");
char[] chars = new char[7];
IOUtils.readFully(reader, chars);
System.out.println(new String(chars)); //1-Will-

With Offsets and Length

Similar to how we used offsets and lengths with the read method, we can use the same with readFully method as well. Below code is self-explanatory.

InputStream inputStream = new FileInputStream(".../data/sample-file.txt");
bytes = new byte[7];
IOUtils.readFully(inputStream, bytes, 1, 5);
System.out.println(new String(bytes)); //1-Wil

InputStream inputStream = new FileInputStream(".../data/sample-file.txt");
bytes = new byte[70];
IOUtils.readFully(inputStream, bytes, 1, 50); //java.io.EOFException: Length to read: 50 actual: 31

Reader reader = new FileReader(".../data/sample-file.txt");
chars = new char[70];
IOUtils.readFully(reader, chars, 0, 50); // java.io.EOFException: Length to read: 50 actual: 31

Reading lines

The readLines method reads the content of an input stream or a reader as a List<String> (each line will be an entry in the list).

InputStream inputStream = new FileInputStream("../data/sample-file.txt");
List<String> lines = IOUtils.readLines(inputStream, Charset.forName("UTF-8"));
//or IOUtils.readLines(inputStream, "UTF-8");
System.out.println(lines);

Reader reader = new FileReader(".../data/sample-file.txt");
lines = IOUtils.readLines(reader);
System.out.println(lines);

The above prints the file contents as it is.

Writing using IOUtils

Writing into an OutputStream

To write a string, we pass it along with the output stream to which we want to write and the character encoding.

OutputStream outputStream = new FileOutputStream(".../data/sample-output-file.txt");
IOUtils.write("data1", outputStream, StandardCharsets.UTF_8);
IOUtils.write(System.lineSeparator(), outputStream, StandardCharsets.UTF_8);
IOUtils.write("data2", outputStream, StandardCharsets.UTF_8);
outputStream.close();

The file sample-output-file.txt has the below contents.

data1
data2

Writing into a Writer

To write a string, pass the string and the writer instance to the write method. Here we are using a FileWriter.

.Writer writer = new FileWriter(".../data/sample-output-file.txt");
IOUtils.write("data1", writer);
IOUtils.write(System.lineSeparator(), writer);
IOUtils.write("data2", writer);
writer.close();

Other variations

There are lots of other variations of the write method which accepts char[], byte[] or a general CharSequence. All these accept either an OutputStream or a Writer as well.

Copying using IOUtils

The copy method copies from an InputStream/Reader to an OutputStream/Writer. It returns the number of bytes/characters copied. Large streams (over 2B) will return the bytes/chars copied as -1 as we cannot represent the correct number of bytes/chars copied in an int. Use the copyLarge method for copying data over 2GB. 

InputStream inputStream = new FileInputStream(".../data/sample-file.txt");
OutputStream outputStream = new FileOutputStream(".../data/sample-output-file.txt");
int numberOfBytesCopied = IOUtils.copy(inputStream, outputStream);
System.out.println(numberOfBytesCopied);//31
outputStream.close();

The above code copies the data in the source input stream to the output stream. After running this, the target file will have the same data as the source.

Using a reader,
Reader reader = new FileReader(".../data/sample-file.txt");
Writer writer = new FileWriter(".../data/sample-output-file.txt", false);
int numberOfCharsRead = IOUtils.copy(reader, writer);
System.out.println(numberOfCharsRead);//31
writer.close();

There are other variations (combinations) of the copy method to copy from  InputStream/Reader to OutputStream/Writer/Appendable.

Copying data over 2GB

If the stream is over 2B, the recommendation is to use the copyLarge method. The code will look similar to the copy method except that this returns a long to denote the number of bytes/chars copied.

InputStream inputStream = new FileInputStream(".../data/sample-file.txt");
OutputStream outputStream = new FileOutputStream(".../data/sample-output-file.txt");
long numberOfBytesCopied = IOUtils.copyLarge(inputStream, outputStream);
System.out.println(numberOfBytesCopied); //31
outputStream.close();

Reader reader = new FileReader(".../data/sample-file.txt");
Writer writer = new FileWriter(".../data/sample-output-file.txt");
long numberOfCharsCopied = IOUtils.copyLarge(reader, writer);
System.out.println(numberOfBytesCopied); //31
writer.close();

Skip and SkipFully

The skip method skips bytes or characters from the input byte stream or character stream.

InputStream inputStream = new FileInputStream(".../data/sample-file.txt");
OutputStream outputStream = new FileOutputStream(".../data/sample-output-file.txt");
long numBytesSkipped = IOUtils.skip(inputStream, 5);
System.out.println(numBytesSkipped);//5
//IOUtils.copy(inputStream, outputStream);

In the above code, we skip 5 bytes and copy the rest into a new file (into the output stream created out of it). The sample-output-file has the below data.

l-90
2-James-87
3-Kyle-67

Shown below is when using a Reader but passing a value to skip that is more than the number of characters available. In that case, nothing is left in the input stream to be written.

Reader reader = new FileReader(".../data/sample-file.txt");
Writer writer = new FileWriter(".../data/sample-output-file.txt");
long numBytesSkipped = IOUtils.skip(reader, 500);
System.out.println(numBytesSkipped);//31
IOUtils.copy(reader, writer); //empty
writer.close();

As you would have guessed skipFully is strict i.e., it has to skip the requested number of bytes or characters. It would fail if there are not enough left.

Reader reader = new FileReader(".../data/sample-file.txt");
IOUtils.skipFully(reader, 50);//java.io.EOFException: Chars to skip:50 actual: 31

There are only 31 characters in the input stream, but since we asked it to skip 50, it resulted in an EOFException.

IOUtils contentEquals

This method compares the contents of two streams and determines if they are equal or not.
Let us create a new file sample-file-copy.txt which has the same content as sample-file.txt.

InputStream inputStream1 = new FileInputStream(".../data/sample-file.txt");
InputStream inputStream2 = new FileInputStream(".../data/sample-file-copy.txt");
System.out.println(IOUtils.contentEquals(inputStream1, inputStream2));//true


Reader reader1 = new FileReader(".../data/sample-file.txt");
Reader reader2 = new FileReader(".../data/sample-file-copy.txt");
System.out.println(IOUtils.contentEquals(reader1, reader2));//true

We create two Input streams and two readers from the two files. Since they have the same data, contentEquals returned true.

contentEqualsIgnoreEOL

Now let us create a file called sample-file-copy-with-new-lines that has new lines (shown below).

1-Will-90


2-James-87
3-Kyle-67

If we use contentEquals on a stream (or reader) created from this file with the stream (or reader) created from sample-file.txt, it will return false. There is a method called contentEqualsEOL that compares two streams or readers ignoring EOL characters.

Reader reader3 = new FileReader(".../data/sample-file-copy-with-new-lines.txt");
System.out.println(IOUtils.contentEquals(reader1, reader3));//false
System.out.println(IOUtils.contentEqualsIgnoreEOL(reader1, reader3));//true

IOUtils – Miscellaneous methods

toString

Converts the contents of an input stream or a reader to a String.

InputStream inputStream = new FileInputStream(".../data/sample-file.txt");
System.out.println(IOUtils.toString(inputStream, StandardCharsets.UTF_8));

Reader reader = new FileReader(".../data/sample-file.txt");
System.out.println(IOUtils.toString(reader));

toInputStream

This is the inverse of toString. This converts the string to an input stream, encoded as bytes using the specified character encoding.

String s = "some-data";
InputStream generatedInputStream = IOUtils.toInputStream(s, StandardCharsets.UTF_8);

toCharArray

Gets an input stream or a reader as a character array.

InputStream inputStream = new FileInputStream(".../data/sample-file.txt");
char[] chars = IOUtils.toCharArray(inputStream, StandardCharsets.UTF_8);
System.out.println(new String(chars));

Reader reader = new FileReader(".../data/sample-file.txt");
chars = IOUtils.toCharArray(reader);
System.out.println(new String(chars));

toByteArray

Gets an input stream or a reader as a byte array.

InputStream inputStream = new FileInputStream(".../data/sample-file.txt");
byte[] bytes = IOUtils.toByteArray(inputStream);
System.out.println(new String(bytes));

Reader reader = new FileReader(".../data/sample-file.txt");
bytes = IOUtils.toByteArray(reader, StandardCharsets.UTF_8);
System.out.println(new String(bytes));

IOUtils closeQuietly

This method closes an OutputStream unconditionally. It ignores any exceptions from the output stream’s close call.

OutputStream outputStream = null;
try {
    outputStream = new FileOutputStream(".../data/sample-output-file.txt");
    //use the outputStream
} catch (IOException e) {
    // error handling
} finally {
    IOUtils.closeQuietly(outputStream);
}

But as of Apache Commons IO 2.6, this method is deprecated.

Conclusion

We learnt about the Apache Commons IO’s IOUtils in this post. We learnt about the read, copy and write methods which allows us to read, copy and write with input/output streams, readers and writers. Then we looked at skip and skipFully methods, the contentEquals methods and few others too.
Refer to the other posts on Apache Commons libraries.

References

Tags: ,

Leave a Reply