When solving algorithm problems, you inevitably end up handling input values. To cover various cases, you take user input directly and process it. When you do, nine times out of ten, the following situation comes up. To send a dataset, a collection of data is joined into a single string using a delimiter (space or comma).

$$[ "A", "B", "C", "D" ] -> "A B C D"$$

For example, to pass an array like $[ "A", "B", "C", "D" ]$ as shown above, each element is separated by a space and delivered as $"A B C D"$. Personally, I usually use the split method, but while looking through algorithm solutions, I came across quite a few pieces of code using a class called StringTokenizer. Since it was a class I'd never seen before, and there must be some reason to deliberately use it in place of the far more accessible split, I decided to directly compare their performance. Execution speed is also an important metric in algorithms, so it's worth shaving off even a little bit of time. Unfortunately, my code optimization skills are terrible, so I need to trim these small details wherever I can. Not bothering to optimize the core logic and instead trying to save time on things like this does feel a bit like drinking diet soda while eating pizza to lose weight, but if StringTokenizer really does perform better, it would be worth applying to algorithm problems going forward.

The split method is the traditional method for splitting a string by a specific delimiter. It's a keyword that exists in many languages besides JAVA — C(++, #), JavaScript, Python, and so on — so it's the first thing most people try when they need to split a string in any language.

JAVA's split is a method included in the String class, the data type for string data. If you have string data, you can call split to split the string. It returns a String[] object.

Usage is as follows.

JAVA
import java.util.Arrays;

/**
 * 메인 클래스
 *
 * @author RWB
 * @since 2021.06.13 Sun 22:50:57
 */
public class Main
{
	/**
	 * 메인 함수
	 *
	 * @param args: [String[]] 매개변수
	 */
	public static void main(String[] args)
	{
		String text = "A B C D";
		
		String[] splited = text.split(" ");
		
		System.out.println(Arrays.toString(splited));
	}
}

The output is as follows.

TC
[A, B, C, D]

You can see that the string A B C D is split into [A, B, C, D] based on spaces. There's one other interesting point: JAVA's split method accepts a regular expression as the delimiter. If used well, this lets you use compound delimiters.

The direct reason I ended up writing this post. StringTokenizer is also a class specialized for splitting strings. Unlike split, which returns a String[], the difference is that it's its own standalone class.

You initialize and use it in a form like StringTokenizer tokenizer = new StringTokenizer("string");. Here are some useful methods to know when working with a StringTokenizer instance.

You can also add a delimiter as an argument to the constructor, like StringTokenizer tokenizer = new StringTokenizer("string", "delimiter");, to split on a delimiter of your choosing. If not specified separately, the default delimiter is \t\n\r\t, which splits on line breaks, spaces, and tabs. There's one thing to watch out for here: the default delimiter \t\n\r\t includes line breaks, spaces, and tabs all at once. In other words, if spaces and line breaks are mixed together, as in A B C D\nA B C D, both spaces and line breaks are split on, producing output like [A, B, C, D, A, B, C, D]. If you explicitly specify a delimiter in the constructor, you can prevent this. When you specify it directly, you can also use multiple characters, not just spaces or line breaks.

JAVA
import java.util.Arrays;
import java.util.StringTokenizer;

/**
 * 메인 클래스
 *
 * @author RWB
 * @since 2021.06.13 Sun 23:48:14
 */
public class Test
{
	/**
	 * 메인 함수
	 *
	 * @param args: [String[]] 매개변수
	 */
	public static void main(String[] args)
	{
		String text = "A B C D";
		
		StringTokenizer tokenizer = new StringTokenizer(text);
		
		String[] splited = new String[tokenizer.countTokens()];
		
		for (int i = 0; i < splited.length; i++)
		{
			splited[i] = tokenizer.nextToken();
		}
		
		System.out.println(Arrays.toString(splited));
	}
}

The output is the same.

TC
[A, B, C, D]

So how do split and StringTokenizer compare in terms of performance? I wrote a simple test program to compare them.

1. Assign a repeat count t.
2. For each case, generate a random string of 5 to 20 characters. Each character is separated by a space.
3. Split the string using a space as the delimiter
   1. Using split
   2. Using StringTokenizer
4. Calculate the total elapsed time and average elapsed time
5. Display the results

The source is as follows.

JAVA
import java.text.DecimalFormat;
import java.util.Arrays;
import java.util.Random;
import java.util.StringTokenizer;

/**
 * 메인 클래스
 *
 * @author RWB
 * @since 2021.06.14 Mon 00:06:32
 */
public class Main
{
	/**
	 * 메인 함수
	 *
	 * @param args: [String[]] 매개변수
	 */
	public static void main(String[] args)
	{
		int t = 10000;
		
		long[] timer = { 0, 0 };
		
		int[] sum = { 0, 0 };
		
		for (int i = 0; i < t; i++)
		{
			int random = (int) ((Math.random() * (20 - 5)) + 5);
			
			String text = getTestString(random);
			
			// split 로직 ----------------------------------------
			long timeStart = System.nanoTime();
			
			String[] a1 = useSplit(text);
			
			long timeEnd = System.nanoTime() - timeStart;
			
			sum[0] += a1.length;
			
			timer[0] += timeEnd;
			
			System.out.println(Arrays.toString(a1) + ": " + addComma(timeEnd) + "ns");
			// split 로직 ----------------------------------------
			
			// StringTokenizer 로직 ----------------------------------------
			timeStart = System.nanoTime();
			
			String[] a2 = useStringTokenizer(text);
			
			timeEnd = System.nanoTime() - timeStart;
			
			sum[1] += a2.length;
			
			timer[1] += timeEnd;
			
			System.out.println(Arrays.toString(a2) + ": " + addComma(timeEnd) + "ns");
			// StringTokenizer 로직 ----------------------------------------
		}
		
		System.out.println(addComma(t) + "개 데이터 그룹 수행");
		
		System.out.println();
		
		System.out.println("split 결과");
		System.out.println(" * 총 소요: " + addComma(timer[0]) + "ns");
		System.out.println(" * 평균 소요: " + addComma((timer[0] / t)) + "ns");
		System.out.println(" * 분해한 요소: " + addComma(sum[0]) + "개");
		
		System.out.println();
		
		System.out.println("StringTokenizer 결과");
		System.out.println(" * 총 소요: " + addComma(timer[1]) + "ns");
		System.out.println(" * 평균 소요: " + addComma((timer[1] / t)) + "ns");
		System.out.println(" * 분해한 요소: " + addComma(sum[1]) + "개");
		
		System.out.println();
		
		System.out.println("split " + (timer[0] == timer[1] ? "==" : (timer[0] > timer[1]) ? "<" : ">") + " StringTokenizer");
	}
	
	/**
	 * 구분된 문자열 반환 함수 (split)
	 *
	 * @param text: [String] 대상 문자열
	 *
	 * @return [String[]] 구분된 문자열
	 */
	private static String[] useSplit(String text)
	{
		return text.split(" ");
	}
	
	/**
	 * 구분된 문자열 반환 함수 (StringTokenizer)
	 *
	 * @param text: [String] 대상 문자열
	 *
	 * @return [String[]] 구분된 문자열
	 */
	private static String[] useStringTokenizer(String text)
	{
		StringTokenizer tokenizer = new StringTokenizer(text, " ");
		
		int count = tokenizer.countTokens();
		
		String[] result = new String[count];
		
		for (int i = 0; i < count; i++)
		{
			result[i] = tokenizer.nextToken();
		}
		
		return result;
	}
	
	/**
	 * 무작위 문자열 반환 함수
	 *
	 * @param n: [int] 문자 갯수
	 *
	 * @return [String] 무작위 문자
	 */
	private static String getTestString(int n)
	{
		Random random = new Random();
		
		StringBuilder builder = new StringBuilder();
		
		for (int i = 0; i < n; i++)
		{
			builder.append((char) ((random.nextInt(26)) + 97)).append(" ");
		}
		
		return builder.toString().trim();
	}
	
	/**
	 * 1000 단위 구분 숫자 반환 함수
	 *
	 * @param num: [long] 대상 숫자
	 *
	 * @return [String] 1000 단위 구분 숫자
	 */
	private static String addComma(long num)
	{
		DecimalFormat format = new DecimalFormat(",###");
		
		return format.format(num);
	}
}

The results of running it 10 times per count are summarized in the tables below.

• $t = 1$

With a single iteration, StringTokenizer wins decisively, 8 to 2.

• $t = 100$

At 100 iterations, StringTokenizer again wins decisively, 10 to 0.

• $t = 1,000$

The reason I jumped from clean powers like $2^1$, $2^3$ straight to 1000 out of nowhere is that, oddly enough, split wins decisively at $t = 1,000$.

• $t = 10,000$

StringTokenizer wins decisively again, 8 to 2.

• $t = 1,000,000$

StringTokenizer wins decisively, 8 to 2.

Except for the peculiar case of $t = 1,000$, StringTokenizer generally performs better. I don't fully understand why that particular case behaves that way. Of course, since statistics become more meaningful as the sample size grows, it's hard to draw firm conclusions from just 10 runs.

On my work computer (AMD Ryzen 2700X), StringTokenizer was faster in every single case. There may be some variation in the computation results or behavior depending on the CPU.

According to the JAVA API, StringTokenizer is a legacy class kept around for backward compatibility. The JAVA API recommends using split or the regex package instead of StringTokenizer wherever possible.

Original text
StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead.

Based on the numbers in the tables, StringTokenizer is up to about 20% faster than split. However, the JAVA API recommends using alternatives wherever possible, and even after a million operations, the difference stays in the millisecond range. So while there's a relative difference, in objective terms there's not much difference at all. Rather than reaching for a whole new class just to split strings, I think it's more efficient to just use split, which operates directly on the string itself.