Multithreading and Concurrency

1. What do we understand by the term concurrency?

Concurrency is the ability of a program to execute several computations simultaneously. This can be achieved by distributing the computations over the available CPU cores of a machine or even over different machines within the same network.

2. What is the difference between processes and threads?

A process is an execution environment provided by the operating system that has its own set of private resources (e.g. memory, open files, etc.). Threads, in contrast to processes, live within a process and share their resources (memory, open files, etc.) with the other threads of the process. The ability to share resources between different threads makes thread more suitable for tasks where performance is a significant requirement.

3. In Java, what is a process and a thread?

In Java, processes correspond to a running Java Virtual Machine (JVM) whereas threads live within the JVM and can be created and stopped by the Java application dynamically at runtime.

4. What is a scheduler?

A scheduler is the implementation of a scheduling algorithm that manages access of processes and threads to some limited resource like the processor or some I/O channel. The goal of most scheduling algorithms is to provide some kind of load balancing for the available processes/threads that guarantees that each process/thread gets an appropriate time frame to access the requested resource exclusively.

5. How many threads does a Java program have at least?

Each Java program is executed within the main thread; hence each Java application has at least one thread.

6. How can a Java application access the current thread?

The current thread can be accessed by calling the static method currentThread() of the JDK class java.lang.Thread:

	public class MainThread {

		public static void main(String[] args) {
			long id = Thread.currentThread().getId();
			String name = Thread.currentThread().getName();
			...
		}
	}

7. What properties does each Java thread have?

Each Java thread has the following properties:

  • an identifier of type long that is unique within the JVM
  • a name of type String
  • a priority of type int
  • a state of type java.lang.Thread.State
  • a thread group the thread belongs to

8. What is the purpose of thread groups?

Each thread belongs to a group of threads. The JDK class java.lang.ThreadGroup provides some methods to handle a whole group of Threads. With these methods we can, for example, interrupt all threads of a group or set their maximum priority.

9. What states can a thread have and what is the meaning of each state?

  • NEW: A thread that has not yet started is in this state.
  • RUNNABLE: A thread executing in the Java virtual machine is in this state.
  • BLOCKED: A thread that is blocked waiting for a monitor lock is in this state.
  • WAITING: A thread that is waiting indefinitely for another thread to perform a particular action is in this state.
  • TIMED_WAITING: A thread that is waiting for another thread to perform an action for up to a specified waiting time is in this state.
  • TERMINATED: A thread that has exited is in this state.

10. How do we set the priority of a thread?

The priority of a thread is set by using the method setPriority(int). To set the priority to the maximum value, we use the constant Thread.MAX_PRIORITY and to set it to the minimum value we use the constant Thread.MIN_PRIORITY because these values can differ between different JVM implementations.

11. How is a thread created in Java?

Basically, there are two ways to create a thread in Java.

The first one is to write a class that extends the JDK class java.lang.Thread and call its method start():

	public class MyThread extends Thread {

		public MyThread(String name) {
			super(name);
		}

		@Override
		public void run() {
			System.out.println("Executing thread "+Thread.currentThread().getName());
		}

		public static void main(String[] args) throws InterruptedException {
			MyThread myThread = new MyThread("myThread");
			myThread.start();
		}
	}

The second way is to implement the interface java.lang.Runnable and pass this implementation as a parameter to the constructor of java.lang.Thread:

	public class MyRunnable implements Runnable {

		public void run() {
			System.out.println("Executing thread "+Thread.currentThread().getName());
		}

		public static void main(String[] args) throws InterruptedException {
			Thread myThread = new Thread(new MyRunnable(), "myRunnable");
			myThread.start();
		}
	}

12. How do we stop a thread in Java?

To stop a thread one can use a volatile reference pointing to the current thread that can be set to null by other threads to indicate the current thread should stop its execution:

	private static class MyStopThread extends Thread {
		private volatile Thread stopIndicator;

		public void start() {
			stopIndicator = new Thread(this);
			stopIndicator.start();
		}

		public void stopThread() {
			stopIndicator = null;
		}

		@Override
		public void run() {
			Thread thisThread = Thread.currentThread();
			while(thisThread == stopIndicator) {
				try {
					Thread.sleep(1000);
				} catch (InterruptedException e) {
				}
			}
		}
	}

13. Why should a thread not be stopped by calling its method stop()?

A thread should not be stopped by using the deprecated methods stop() of java.lang.Thread, as a call of this method causes the thread to unlock all monitors it has acquired. If any object protected by one of the released locks was in an inconsistent state, this state gets visible to all other threads. This can cause arbitrary behavior when other threads work this this inconsistent object.

14. Is it possible to start a thread twice?

No, after having started a thread by invoking its start() method, a second invocation of start() will throw an IllegalThreadStateException.

15. What is the output of the following code?

	public class MultiThreading {

		private static class MyThread extends Thread {

			public MyThread(String name) {
				super(name);
			}

			@Override
			public void run() {
				System.out.println(Thread.currentThread().getName());
			}
		}

		public static void main(String[] args) {
			MyThread myThread = new MyThread("myThread");
			myThread.run();
		}
	}

The code above produces the output “main” and not “myThread”. As can be seen in line two of the main() method, we invoke by mistake the method run() instead of start(). Hence, no new thread is started, but the method run() gets executed within the main thread.

16. What is a daemon thread?

A daemon thread is a thread whose execution state is not evaluated when the JVM decides if it should stop or not. The JVM stops when all user threads (in contrast to the daemon threads) are terminated. Hence daemon threads can be used to implement for example monitoring functionality as the thread is stopped by the JVM as soon as all user threads have stopped:

	public class Example {

		private static class MyDaemonThread extends Thread {

			public MyDaemonThread() {
				setDaemon(true);
			}

			@Override
			public void run() {
				while (true) {
					try {
						Thread.sleep(1);
					} catch (InterruptedException e) {
						e.printStackTrace();
					}
				}
			}
		}

		public static void main(String[] args) throws InterruptedException {
			Thread thread = new MyDaemonThread();
			thread.start();
		}
	}

The example application above terminates even though the daemon thread is still running in its endless while loop.

17. Is it possible to convert a normal user thread into a daemon thread after it has been started?

A user thread cannot be converted into a daemon thread once it has been started. Invoking the method thread.setDaemon(true) on an already running thread instance causes a IllegalThreadStateException.

18. What do we understand by busy waiting?

Busy waiting means implementations that wait for an event by performing some active computations that let the thread/process occupy the processor although it could be removed from it by the scheduler. An example for busy waiting would be to spend the waiting time within a loop that determines the current time again and again until a certain point in time is reached:

	Thread thread = new Thread(new Runnable() {
		@Override
		public void run() {
			long millisToStop = System.currentTimeMillis() + 5000;
			long currentTimeMillis = System.currentTimeMillis();
			while (millisToStop > currentTimeMillis) {
				currentTimeMillis = System.currentTimeMillis();
			}
		}
	});

19. How can we prevent busy waiting?

One way to prevent busy waiting is to put the current thread to sleep for a given amount of time. This can be done by calling the method java.lang.Thread.sleep(long) by passing the number of milliseconds to sleep as an argument.

20. Can we use Thread.sleep() for real-time processing?

The number of milliseconds passed to an invocation of Thread.sleep(long) is only an indication for the scheduler how long the current thread does not need to be executed. It may happen that the scheduler lets the thread execute again a few milliseconds earlier or later depending on the actual implementation. Hence an invocation of Thread.sleep() should not be used for real-time processing.

21. How can a thread be woken up that has been put to sleep before using Thread.sleep()?

The method interrupt() of java.lang.Thread interrupts a sleeping thread. The interrupted thread that has been put to sleep by calling Thread.sleep() is woken up by an InterruptedException:

	public class InterruptExample implements Runnable {

		public void run() {
			try {
				Thread.sleep(Long.MAX_VALUE);
			} catch (InterruptedException e) {
				System.out.println("["+Thread.currentThread().getName()+"] Interrupted by exception!");
			}
		}

		public static void main(String[] args) throws InterruptedException {
			Thread myThread = new Thread(new InterruptExample(), "myThread");
			myThread.start();

			System.out.println("["+Thread.currentThread().getName()+"] Sleeping in main thread for 5s...");
			Thread.sleep(5000);

			System.out.println("["+Thread.currentThread().getName()+"] Interrupting myThread");
			myThread.interrupt();
		}
	}

22. How can a thread query if it has been interrupted?

If the thread is not within a method like Thread.sleep() that would throw an InterruptedException, the thread can query if it has been interrupted by calling either the static method Thread.interrupted() or the method isInterrupted() that it has inherited from java.lang.Thread.

23. How should an InterruptedException be handled?

Methods like sleep() and join() throw an InterruptedException to tell the caller that another thread has interrupted this thread. In most cases this is done in order to tell the current thread to stop its current computations and to finish them unexpectedly. Hence ignoring the exception by catching it and only logging it to the console or some log file is often not the appropriate way to handle this kind of exception. The problem with this exception is, that the method run() of the Runnable interface does not allow that run() throws any exceptions. So just rethrowing it does not help. This means the implementation of run() has to handle this checked exception itself and this often leads to the fact that it its caught and ignored.

24. After having started a child thread, how do we wait in the parent thread for the termination of the child thread?

Waiting for a thread’s termination is done by invoking the method join() on the thread’s instance variable:

	Thread thread = new Thread(new Runnable() {
		@Override
		public void run() {

		}
	});
	thread.start();
	thread.join();

25. What is the output of the following program?

public class MyThreads {

	private static class MyDaemonThread extends Thread {

		public MyDaemonThread() {
			setDaemon(true);
		}

		@Override
		public void run() {
			try {
				Thread.sleep(1000);
			} catch (InterruptedException e) {
			}
		}
	}

	public static void main(String[] args) throws InterruptedException {
		Thread thread = new MyDaemonThread();
		thread.start();
		thread.join();
		System.out.println(thread.isAlive());
	}
}

The output of the above code is “false”. Although the instance of MyDaemonThread is a daemon thread, the invocation of join() causes the main thread to wait until the execution of the daemon thread has finished. Hence calling isAlive() on the thread instance reveals that the daemon thread is no longer running.

26. What happens when an uncaught exception leaves the run() method?

I can happen that an unchecked exception escapes from the run() method. In this case the thread is stopped by the Java Virtual Machine. It is possible to catch this exception by registering an instance that implements the interface UncaughtExceptionHandler as an exception handler.

This is either done by invoking the static method Thread.setDefaultUncaughtExceptionHandler(Thread.UncaughtExceptionHandler), which tells the JVM to use the provided handler in case there was no specific handler registerd on the thread itself, or by invoking setUncaughtExceptionHandler(Thread.UncaughtExceptionHandler) on the thread instance itself.

27. What is a shutdown hook?

A shutdown hook is a thread that gets executed when the JVM shuts down. It can be registered by invoking addShutdownHook(Runnable) on the Runtime instance:

	Runtime.getRuntime().addShutdownHook(new Thread() {
		@Override
		public void run() {

		}
	});

28. For what purposes is the keyword synchronized used?

When you have to implement exclusive access to a resource, like some static value or some file reference, the code that works with the exclusive resource can be embraced with a synchronized block:

	synchronized (SynchronizedCounter.class) {
		counter++;
	}

29. What intrinsic lock does a synchronized method acquire?

A synchronized method acquires the intrinsic lock for that method’s object and releases it when the method returns. Even if the method throws an exception, the intrinsic lock is released. Hence a synchronized method is equal to the following code:

	public void method() {
		synchronized(this) {
			...
		}
	}

30. Can a constructor be synchronized?

No, a constructor cannot be synchronized. The reason why this leads to an syntax error is the fact that only the constructing thread should have access to the object being constructed.

31. Can primitive values be used for intrinsic locks?

No, primitive values cannot be used for intrinsic locks.

32. Are intrinsic locks reentrant?

Yes, intrinsic locks can be accessed by the same thread again and again. Otherwise code that acquires a lock would have to pay attention that it does not accidently tries to acquire a lock it has already acquired.

33. What do we understand by an atomic operation?

An atomic operation is an operation that is either executed completely or not at all.

34. Is the statement c++ atomic?

No, the incrementation of an integer variable consist of more than one operation. First we have to load the current value of c, increment it and then finally store the new value back. The current thread performing this incrementation may be interrupted in-between any of these three steps, hence this operation is not atomic.

35. What operations are atomic in Java?

The Java language provides some basic operations that are atomic and that therefore can be used to make sure that concurrent threads always see the same value:

  • Read and write operations to reference variables and primitive variables (except long and double)
  • Read and write operations for all variables declared as volatile

36. Is the following implementation thread-safe?

	public class DoubleCheckedSingleton {
		private DoubleCheckedSingleton instance = null;

		public DoubleCheckedSingleton getInstance() {
			if(instance == null) {
				synchronized (DoubleCheckedSingleton.class) {
					if(instance == null) {
						instance = new DoubleCheckedSingleton();
					}
				}
			}
			return instance;
		}
	}

The code above is not thread-safe. Although it checks the value of instance once again within the synchronized block (for performance reasons), the JIT compiler can rearrange the bytecode in a way that the reference to instance is set before the constructor has finished its execution. This means the method getInstance() returns an object that may not have been initialized completely. To make the code thread-safe, the keyword volatile can be used since Java 5 for the instance variable. Variables that are marked as volatile get only visible to other threads once the constructor of the object has finished its execution completely.

37. What do we understand by a deadlock?

A deadlock is a situation in which two (or more) threads are each waiting on the other thread to free a resource that it has locked, while the thread itself has locked a resource the other thread is waiting on:
Thread 1: locks resource A, waits for resource B
Thread 2: locks resource B, waits for resource A

38. What are the requirements for a deadlock situation?

In general the following requirements for a deadlock can be identified:

  • Mutual exclusion: There is a resource which can be accessed only by one thread at any point in time.
  • Resource holding: While having locked one resource, the thread tries to acquire another lock on some other exclusive resource.
  • No preemption: There is no mechanism, which frees the resource if one thread holds the lock for a specific period of time.
  • Circular wait: During runtime a constellation occurs in which two (or more) threads are each waiting on the other thread to free a resource that it has locked.

39. Is it possible to prevent deadlocks at all?

In order to prevent deadlocks one (or more) of the requirements for a deadlock has to be eliminated:

  • Mutual exclusion: In some situation it is possible to prevent mutual exclusion by using optimistic locking.
  • Resource holding: A thread may release all its exclusive locks, when it does not succeed in obtaining all exclusive locks.
  • No preemption: Using a timeout for an exclusive lock frees the lock after a given amount of time.
  • Circular wait: When all exclusive locks are obtained by all threads in the same sequence, no circular wait occurs.

40. Is it possible to implement a deadlock detection?

When all exclusive locks are monitored and modelled as a directed graph, a deadlock detection system can search for two threads that are each waiting on the other thread to free a resource that it has locked. The waiting threads can then be forced by some kind of exception to release the lock the other thread is waiting on.

41. What is a livelock?

A livelock is a situation in which two or more threads block each other by responding to an action that is caused by another thread. In contrast to a deadlock situation, where two or more threads wait in one specific state, the threads that participate in a livelock change their state in a way that prevents progress on their regular work. An example would be a situation in which two threads try to acquire two locks, but release a lock they have acquired when they cannot acquire the second lock. It may now happen that both threads concurrently try to acquire the first thread. As only one thread succeeds, the second thread may succeed in acquiring the second lock. Now both threads hold two different locks, but as both want to have both locks, they release their lock and try again from the beginning. This situation may now happen again and again.

42. What do we understand by thread starvation?

Threads with lower priority get less time for execution than threads with higher priority. When the threads with lower priority performs a long enduring computations, it may happen that these threads do not get enough time to finish their computations just in time. They seem to “starve” away as threads with higher priority steal them their computation time.

43. Can a synchronized block cause thread starvation?

The order in which threads can enter a synchronized block is not defined. So in theory it may happen that in case many threads are waiting for the entrance to a synchronized block, some threads have to wait longer than other threads. Hence they do not get enough computation time to finish their work in time.

44. What do we understand by the term race condition?

A race condition describes constellations in which the outcome of some multi-threaded implementation depends on the exact timing behavior of the participating threads. In most cases it is not desirable to have such a kind of behavior, hence the term race condition also means that a bug due to missing thread synchronization leads to the differing outcome. A simple example for a race condition is the incrementation of an integer variable by two concurrent threads. As the operation consists of more than one single and atomic operation, it may happen that both threads read and increment the same value. After this concurrent incrementation the amount of the integer variable is not increased by two but only by one.

45. What do we understand by fair locks?

A fair lock takes the waiting time of the threads into account when choosing the next thread that passes the barrier to some exclusive resource. An example implementation of a fair lock is provided by the Java SDK: java.util.concurrent.locks.ReentrantLock. If the constructor with the boolean flag set to true is used, the ReentrantLock grants access to the longest-waiting thread.

46. Which two methods that each object inherits from java.lang.Object can be used to implement a simple producer/consumer scenario?

When a worker thread has finished its current task and the queue for new tasks is empty, it can free the processor by acquiring an intrinsic lock on the queue object and by calling the method wait(). The thread will be woken up by some producer thread that has put a new task into the queue and that again acquires the same intrinsic lock on the queue object and calls notify() on it.

47. What is the difference between notify() and notifyAll()?

Both methods are used to wake up one or more threads that have put themselves to sleep by calling wait(). While notify() only wakes up one of the waiting threads, notifyAll() wakes up all waiting threads.

48. How it is determined which thread wakes up by calling notify()?

It is not specified which threads will be woken up by calling notify() if more than one thread is waiting. Hence code should not rely on any concrete JVM implementation.

49. Is the following code that retrieves an integer value from some queue implementation correct?

	public Integer getNextInt() {
		Integer retVal = null;
		synchronized (queue) {
			try {
				while (queue.isEmpty()) {
					queue.wait();
				}
			} catch (InterruptedException e) {
				e.printStackTrace();
			}
		}
		synchronized (queue) {
			retVal = queue.poll();
			if (retVal == null) {
				System.err.println("retVal is null");
				throw new IllegalStateException();
			}
		}
		return retVal;
	}

Although the code above uses the queue as object monitor, it does not behave correctly in a multi-threaded environment. The reason for this is that it has two separate synchronized blocks. When two threads are woken up in line 6 by another thread that calls notifyAll(), both threads enter one after the other the second synchronized block. It this second block the queue has now only one new value, hence the second thread will poll on an empty queue and get null as return value.

50. Is it possible to check whether a thread holds a monitor lock on some given object?

The class java.lang.Thread provides the static method Thread.holdsLock(Object) that returns true if and only if the current thread holds the lock on the object given as argument to the method invocation.

51. What does the method Thread.yield() do?

An invocation of the static method Thread.yield() gives the scheduler a hint that the current thread is willing to free the processor. The scheduler is free to ignore this hint. As it is not defined which thread will get the processor after the invocation of Thread.yield(), it may even happen that the current thread becomes the “next” thread to be executed.

52. What do you have to consider when passing object instances from one thread to another?

When passing objects between threads, you will have to pay attention that these objects are not manipulated by two threads at the same time. An example would be a Map implementation whose key/value pairs are modified by two concurrent threads. In order to avoid problems with concurrent modifications you can design an object to be immutable.

53. Which rules do you have to follow in order to implement an immutable class?

  • All fields should be final and private.
  • There should be not setter methods.
  • The class itself should be declared final in order to prevent subclasses to violate the principle of immutability.
  • If fields are not of a primitive type but a reference to another object:
    • There should not be a getter method that exposes the reference directly to the caller.
    • Don’t change the referenced objects (or at least changing these references is not visisble to clients of the object).

54. What is the purpose of the class java.lang.ThreadLocal?

As memory is shared between different threads, ThreadLocal provides a way to store and retrieve values for each thread separately. Implementations of ThreadLocal store and retrieve the values for each thread independently such that when thread A stores the value A1 and thread B stores the value B1 in the same instance of ThreadLocal, thread A later on retrieves value A1 from this ThreadLocal instance and thread B retrieves value B1.

55. What are possible use cases for java.lang.ThreadLocal?

Instances of ThreadLocal can be used to transport information throughout the application without the need to pass this from method to method. Examples would be the transportation of security/login information within an instance of ThreadLocal such that it is accessible by each method. Another use case would be to transport transaction information or in general objects that should be accessible in all methods without passing them from method to method.

56. Is it possible to improve the performance of an application by the usage of multi-threading? Name some examples.

If we have more than one CPU core available, the performance of an application can be improved by multi-threading if it is possible to parallelize the computations over the available CPU cores. An example would be an application that should scale all images that are stored within a local directory structure. Instead of iterating over all images one after the other, a producer/consumer implementation can use a single thread to scan the directory structure and a bunch of worker threads that perform the actual scaling operation. Another example would be an application that mirrors some web page. Instead of loading one HTML page after the other, a producer thread can parse the first HTML page and issue the links it found into a queue. The worker threads monitor the queue and load the web pages found by the parser. While the worker threads wait for the page to get loaded completely, other threads can use the CPU to parse the already loaded pages and issue new requests.

57. What do we understand by the term scalability?

Scalability means the ability of a program to improve the performance by adding further resources to it.

58. Is it possible to compute the theoretical maximum speed up for an application by using multiple processors?

Amdahl’s law provides a formula to compute the theoretical maximum speed up by providing multiple processors to an application. The theoretical speedup is computed by S(n) = 1 / (B + (1-B)/n) where n denotes the number of processors and B the fraction of the program that cannot be executed in parallel. When n converges against infinity, the term (1-B)/n converges against zero. Hence the formula can be reduced in this special case to 1/B. As we can see, the theoretical maximum speedup behaves reciprocal to the fraction that has to be executed serially. This means the lower this fraction is, the more theoretical speedup can be achieved.

59. What do we understand by lock contention?

Lock contention occurs, when two or more threads are competing in the acquisition of a lock. The scheduler has to decide whether it lets the thread, which has to wait sleeping and performs a context switch to let another thread occupy the CPU, or if letting the waiting thread busy-waiting is more efficient. Both ways introduce idle time to the inferior thread.

60. Which techniques help to reduce lock contention?

In some cases lock contention can be reduced by applying one of the following techniques:

  • The scope of the lock is reduced.
  • The number of times a certain lock is acquired is reduced (lock splitting).
  • Using hardware supported optimistic locking operations instead of synchronization.
  • Avoid synchronization where possible.
  • Avoid object pooling.

61. Which technique to reduce lock contention can be applied to the following code?

	synchronized (map) {
		UUID randomUUID = UUID.randomUUID();
		Integer value = Integer.valueOf(42);
		String key = randomUUID.toString();
		map.put(key, value);
	}

The code above performs the computation of the random UUID and the conversion of the literal 42 into an Integer object within the synchronized block, although these two lines of code are local to the current thread and do not affect other threads. Hence they can be moved out of the synchronized block:

	UUID randomUUID = UUID.randomUUID();
	Integer value = Integer.valueOf(42);
	String key = randomUUID.toString();
	synchronized (map) {
		map.put(key, value);
	}

62. Explain by an example the technique lock splitting.

Lock splitting may be a way to reduce lock contention when one lock is used to synchronize access to different aspects of the same application. Suppose we have a class that implements the computation of some statistical data of our application. A first version of this class uses the keyword synchronized in each method signature in order to guard the internal state before corruption by multiple concurrent threads. This also means that each method invocation may cause lock contention as other threads may try to acquire the same lock simultaneously. But it may be possible to split the lock on the object instance into a few smaller locks for each type of statistical data within each method. Hence thread T1 that tries to increment the statistical data D1 does not have to wait for the lock while thread T2 simultaneously updates the data D2.

63. What kind of technique for reducing lock contention is used by the SDK class ReadWriteLock?

The SDK class ReadWriteLock uses the fact that concurrent threads do not have to acquire a lock when they want to read a value when no other thread tries to update the value. This is implemented by a pair of locks, one for read-only operations and one for writing operations. While the read-only lock may be obtained by more than one thread, the implementation guarantees that all read operation see an updated value once the write lock is released.

64. What do we understand by lock striping?

In contrast to lock splitting, where we introduce different locks for different aspects of the application, lock striping uses multiple locks to guard different parts of the same data structure. An example for this technique is the class ConcurrentHashMap from JDK’s java.util.concurrent package. The Map implementation uses internally different buckets to store its values. The bucket is chosen by the value’s key. ConcurrentHashMap now uses different locks to guard different hash buckets. Hence one thread that tries to access the first hash bucket can acquire the lock for this bucket, while another thread can simultaneously access a second bucket. In contrast to a synchronized version of HashMap this technique can increase the performance when different threads work on different buckets.

65. What do we understand by a CAS operation?

CAS stands for compare-and-swap and means that the processor provides a separate instruction that updates the value of a register only if the provided value is equal to the current value. CAS operations can be used to avoid synchronization as the thread can try to update a value by providing its current value and the new value to the CAS operation. If another thread has meanwhile updated the value, the thread’s value is not equal to the current value and the update operation fails. The thread then reads the new value and tries again. That way the necessary synchronization is interchanged by an optimistic spin waiting.

66. Which Java classes use the CAS operation?

The SDK classes in the package java.util.concurrent.atomic like AtomicInteger or AtomicBoolean use internally the CAS operation to implement concurrent incrementation.

	public class CounterAtomic {
		private AtomicLong counter = new AtomicLong();

		public void increment() {
			counter.incrementAndGet();
		}

		public long get() {
			return counter.get();
		}
	}

67. Provide an example why performance improvements for single-threaded applications can cause performance degradation for multi-threaded applications.

A prominent example for such optimizations is a List implementation that holds the number of elements as a separate variable. This improves the performance for single-threaded applications as the size() operation does not have to iterate over all elements but can return the current number of elements directly. Within a multi-threaded application the additional counter has to be guarded by a lock as multiple concurrent threads may insert elements into the list. This additional lock can cost performance when there are more updates to the list than invocations of the size() operation.

68. Is object pooling always a performance improvement for multi-threaded applications?

Object pools that try to avoid the construction of new objects by pooling them can improve the performance of single-threaded applications as the cost for object creation is interchanged by requesting a new object from the pool. In multi-threaded applications such an object pool has to have synchronized access to the pool and the additional costs of lock contention may outweigh the saved costs of the additional construction and garbage collection of the new objects. Hence object pooling may not always improve the overall performance of a multi-threaded application.

69. What is the relation between the two interfaces Executor and ExecutorService?

The interface Executor only defines one method: execute(Runnable). Implementations of this interface will have to execute the given Runnable instance at some time in the future. The ExecutorService interface is an extension of the Executor interface and provides additional methods to shut down the underlying implementation, to await the termination of all submitted tasks and it allows submitting instances of Callable.

70. What happens when you submit() a new task to an ExecutorService instance whose queue is already full?

As the method signature of submit() indicates, the ExecutorService implementation is supposed to throw a RejectedExecutionException.

71. What is a ScheduledExecutorService?

The interface ScheduledExecutorService extends the interface ExecutorService and adds method that allow to submit new tasks to the underlying implementation that should be executed a given point in time. There are two methods to schedule one-shot tasks and two methods to create and execute periodic tasks.

72. Do you know an easy way to construct a thread pool with 5 threads that executes some tasks that return a value?

The SDK provides a factory and utility class Executors whose static method newFixedThreadPool(int nThreads) allows the creation of a thread pool with a fixed number of threads (the implementation of MyCallable is omitted):

	public static void main(String[] args) throws InterruptedException, ExecutionException {
		ExecutorService executorService = Executors.newFixedThreadPool(5);
		Future[] futures = new Future[5];
		for (int i = 0; i < futures.length; i++) {
			futures[i] = executorService.submit(new MyCallable());
		}
		for (int i = 0; i < futures.length; i++) {
			Integer retVal = futures[i].get();
			System.out.println(retVal);
		}
		executorService.shutdown();
	}

73. What is the difference between the two interfaces Runnable and Callable?

The interface Runnable defines the method run() without any return value whereas the interface Callable allows the method run() to return a value and to throw an exception.

74. Which are use cases for the class java.util.concurrent.Future?

Instances of the class java.util.concurrent.Future are used to represent results of asynchronous computations whose result are not immediately available. Hence the class provides methods to check if the asynchronous computation has finished, canceling the task and to the retrieve the actual result. The latter can be done with the two get() methods provided. The first get() methods takes no parameter and blocks until the result is available whereas the second get() method takes a timeout parameter that lets the method invocation return if the result does not get available within the given timeframe.

75. What is the difference between HashMap and Hashtable particularly with regard to thread-safety?

The methods of Hashtable are all synchronized. This is not the case for the HashMap implementation. Hence Hashtable is thread-safe whereas HashMap is not thread-safe. For single-threaded applications it is therefore more efficient to use the “newer” HashMap implementation.

76. Is there a simple way to create a synchronized instance of an arbitrary implementation of Collection, List or Map?

The utility class Collections provides the methods synchronizedCollection(Collection), synchronizedList(List) and synchronizedMap(Map) that return a thread-safe collection/list/map that is backed by the given instance.

77. What is a semaphore?

A semaphore is a data structure that maintains a set of permits that have to be acquired by competing threads. Semaphores can therefore be used to control how many threads access a critical section or resource simultaneously. Hence the constructor of java.util.concurrent.Semaphore takes as first parameter the number of permits the threads compete about. Each invocation of its acquire() methods tries to obtain one of the available permits. The method acquire() without any parameter blocks until the next permit gets available. Later on, when the thread has finished its work on the critical resource, it can release the permit by invoking the method release() on an instance of Semaphore.

78. What is a CountDownLatch?

The SDK class CountDownLatch provides a synchronization aid that can be used to implement scenarios in which threads have to wait until some other threads have reached the same state such that all thread can start. This is done by providing a synchronized counter that is decremented until it reaches the value zero. Having reached zero the CountDownLatch instance lets all threads proceed. This can be either used to let all threads start at a given point in time by using the value 1 for the counter or to wait until a number of threads has finished. In the latter case the counter is initialized with the number of threads and each thread that has finished its work counts the latch down by one.

79. What is the difference between a CountDownLatch and a CyclicBarrier?

Both SDK classes maintain internally a counter that is decremented by different threads. The threads wait until the internal counter reaches the value zero and proceed from there on. But in contrast to the CountDownLatch the class CyclicBarrier resets the internal value back to the initial value once the value reaches zero. As the name indicates instances of CyclicBarrier can therefore be used to implement use cases where threads have to wait on each other again and again.

80. What kind of tasks can be solved by using the Fork/Join framework?

The base class of the Fork/Join Framework java.util.concurrent.ForkJoinPool is basically a thread pool that executes instances of java.util.concurrent.ForkJoinTask. The class ForkJoinTask provides the two methods fork() and join(). While fork() is used to start the asynchronous execution of the task, the method join() is used to await the result of the computation. Hence the Fork/Join framework can be used to implement divide-and-conquer algorithms where a more complex problem is divided into a number of smaller and easier to solve problems.

81. Is it possible to find the smallest number within an array of numbers using the Fork/Join-Framework?

The problem of finding the smallest number within an array of numbers can be solved by using a divide-and-conquer algorithm. The smallest problem that can be solved very easily is an array of two numbers as we can determine the smaller of the two numbers directly by one comparison. Using a divide-and-conquer approach the initial array is divided into two parts of equal length and both parts are provided to two instances of RecursiveTask that extend the class ForkJoinTask. By forking the two tasks they get executed and either solve the problem directly, if their slice of the array has the length two, or they again recursively divide the array into two parts and fork two new RecursiveTasks. Finally each task instance returns its result (either by having it computed directly or by waiting for the two subtasks). The root tasks then returns the smallest number in the array.

82. What is the difference between the two classes RecursiveTask and RecursiveAction?

In contrast to RecursiveTask the method compute() of RecursiveAction does not have to return a value. Hence RecursiveAction can be used when the action works directly on some data structure without having to return the computed value.

83. Is it possible to perform stream operations in Java 8 with a thread pool?

Collections provide the method parallelStream() to create a stream that is processed by a thread pool. Alternatively you can call the intermediate method parallel() on a given stream to convert a sequential stream to a parallel counterpart.

84. How can we access the thread pool that is used by parallel stream operations?

The thread pool used for parallel stream operations can be accessed by ForkJoinPool.commonPool(). This way we can query its level of parallelism with commonPool.getParallelism(). The level cannot be changed at runtime but it can be configured by providing the following JVM parameter: -Djava.util.concurrent.ForkJoinPool.common.parallelism=5.

Reference: http://www.javacodegeeks.com/2014/11/multithreading-concurrency-interview-questions-answers.html

Core concepts

SOAP vs.REST

A NordicAPIs infographic

rest vs soap

originORIGIN

origin REST

REST

REST (Representational State Transfer) was Created in 2000 by Roy Fielding in UC, Irvine. Developed in an academic environment, this protocol embraces the philosophy of the open Web.

origin SOAP

SOAP

SOAP (Simple Object Access Protocol), was created in 1998 by Dave Winer et al in collaboration with Microsoft. Developed by a large software company, this protocol addresses the goal of addressing the needs of the enterprise market.

concept_titleBASIC CONCEPT

concept1

REST

Makes data available as resources (nouns), for example “user” or “invoice””

concept2

SOAP

Makes data available as services (verb + noun), for example “getUser” or “PayInvoice”

concept_titlePROS

pros REST

REST

  • Follows the philosophy of the Open Web
  • Relatively easy to implement and maintain
  • Clearly separates client and server implementations
  • Communication isn’t controlled by a single entity
  • Information can be stored by the client to prevent multiple calls
  • Can return data in multiple formats (JSON, XML etc)
pros SOAP

SOAP

  • Follows a formal enterprise approach
  • Works on top of any communication protocol, even asynchronously
  • Information about objects is communicated to clients
  • Security and authorization are part of the protocol
  • Can be fully described using WSDL

cons_titleCONS

cons REST

REST

  • only works on top of the HTTP protocol
  • Hard to enforce authorization and security on top of it
cons SOAP

SOAP

  • Spends a lot of bandwidth communicating metadata
  • Hard to implement and is unpopular among Web and mobile developers
  • Uses only XML

cons_titleWHEN TO USE

concept1

REST

  • When clients and servers operate on a Web environment
  • When information about objects doesn’t need to be communicated to the client
concept1

SOAP

  • When clients need to have access to objects available on servers
  • When you want to enforce a formal contract between client and server

SOAP vs REST when not to useWHEN NOT TO USE

when not to use REST

REST

  • When you need to enforce a strict contract between client and server
  • When performing transactions that involve multiple calls
when not to use SOAP

SOAP

  • When you want the majority of developers to easily use your API
  • When your bandwidth is very limited

REST vs SOAP use casesCOMMON USE CASES

REST use cases

REST

  • Social Media services
  • Social Networks
  • Web Chat services
  • Mobile Services
SOAP use cases

SOAP

  • Financial services
  • Payment gateways
  • Telecommunication services

Reference: http://nordicapis.com/rest-vs-soap-nordic-apis-infographic-comparison/

Why SOAP?

Here are a few reasons you may want to use SOAP.

WS-Security

While SOAP supports SSL (just like REST) it also supports WS-Security which adds some enterprise security features. Supports identity through intermediaries, not just point to point (SSL). It also provides a standard implementation of data integrity and data privacy. Calling it “Enterprise” isn’t to say it’s more secure, it simply supports some security tools that typical internet services have no need for, in fact they are really only needed in a few “enterprise” scenarios.

WS-AtomicTransaction

Need ACID Transactions over a service, you’re going to need SOAP. While REST supports transactions, it isn’t as comprehensive and isn’t ACID compliant. Fortunately ACID transactions almost never make sense over the internet. REST is limited by HTTP itself which can’t provide two-phase commit across distributed transactional resources, but SOAP can. Internet apps generally don’t need this level of transactional reliability, enterprise apps sometimes do.

WS-ReliableMessaging

Rest doesn’t have a standard messaging system and expects clients to deal with communication failures by retrying. SOAP has successful/retry logic built in and provides end-to-end reliability even through SOAP intermediaries.

WS-AtomicTransaction Explained

The following figure shows two instances of WebLogic Server interacting within the context of a Web services atomic transaction. For simplicity, two WebLogic Web service applications are shown.

Figure 3-2 Web Services Atomic Transactions in WebLogic Server Environment

Description of Figure 3-2 follows
Description of “Figure 3-2 Web Services Atomic Transactions in WebLogic Server Environment”

Please note the following:

  • Using the local JTA transaction manager, a transaction can be imported to or exported from the local JTA environment as a subordinate transaction, all within the context of a Web service request.
  • Creation and management of the coordination context is handled by the local JTA transaction manager.
  • All transaction integrity management and recovery processing is done by the local JTA transaction manager.

For more information about JTA, see Programming JTA for Oracle WebLogic Server.

The following describes a sample end-to-end Web services atomic transaction interaction, illustrated in Figure 3-2:

  1. Application A begins a transaction on the current thread of control using the JTA transaction manager on Server A.
  2. Application A calls a Web service method in Application B on Server B.
  3. Server A updates its transaction information and creates a SOAP header that contains the coordination context, and identifies the transaction and local coordinator.
  4. Server B receives the request for Application B, detects that the header contains a transaction coordination context and determines whether it has already registered as a participant in this transaction. If it has, that transaction is resumed and if not, a new transaction is started.

    Application B executes within the context of the imported transaction. All transactional resources with which the application interacts are enlisted with this imported transaction.

  5. Server B enlists itself as a participant in the WS-AtomicTransaction transaction by registering with the registration service indicated in the transaction coordination context.
  6. Server A resumes the transaction.
  7. Application A resumes processing and commits the transaction.

 


Spring Framework Overview

Spring was built on top of the idea of dependency injection and inversion of control. In normal words – instead of having a bunch of classes creating each other and passing each other from one place to another you have a bag of beans. Each bean declares its dependencies (what services do I need to work?) and Spring container resolves this requirements by automatically and automagically wiring everything together.

You have a Service that says (through XML, annotations, constructor signature…) I need DAO interface to work! and Spring is kind enough to find some bean that implements that interface, create it first and pass where it is required.

On that foundation multiple other services were provided (mostly in terms of data access and AOP), but the injection is the core concept.


Spring Framework – Dependency Injection

Any application is composed of many objects that collaborate with each other to perform some useful stuff. Traditionally each object is responsible for obtaining its own references to the dependent objects (dependencies) it collaborate with. This leads to highly coupled classes and hard-to-test code.

For example,consider a Car object.

A Car depends on Wheels, Engine, Fuel, Battery, etc to run. Traditionally we define the brand of such dependent objects along with the definition of the Car object.

Without Dependency Injection (DI) :

class Car{
  private Wheel wh= new NepaliRubberWheel();
  private Battery bt= new ExcideBattery();
  //rest
}

Here, the Car object is responsible for creating the dependent objects.

What if we want to change the type of its dependent object – say Wheel – after the initial NepaliRubberWheel() punctures? We need to recreate the Car object with its new dependency sayChineseRubberWheel(), but only the Car manufacturer can do that.

Then what the Dependency Injection does us for …?

When using Dependency Injection, objects are given their dependencies at run time rather than compile time (car manufacturing time). So that we can now change the Wheel whenever we want. Here, the Dependency (Wheel) can be injected into Car at run time.

After using DI:

class Car{
  private Wheel wh= [Inject an Instance of Wheel at runtime]
  private Battery bt= [Inject an Instance of Battery at runtime]
}

Source: understanding-dependency-injection


Spring Framework – Aspect Oriented Programming (AOP)

Laymans terms so let me give you an example. Let’s say you have a web application, and you need to add error logging / auditing. One implementation would be to go into every public method and add your try catch blocks etc…

Well Aspect oriented says hogwash with that, let me inject my method around your method so for example instead of calling YourClass.UpdateModel(), the system might call,

LoggingHandler.CallMethod() this method might then redirect the call to UpdateModel but wraps it in a try catch block to handle logging errors.

Now the trick is that this redirection happens automagically, through configuration or by applying attributes to methods.

This works for as you said cross cutting things which are very common programing elements that exist in every domain, such as: Logging, Auditing, Transaction Mgmt, Authorization.

The idea behind it is to remove all this common plumbing code out of your business / app tier so you can focus on solving the problem not worrying about logging this method call or that method call.

Scalable NIO Servers – Performance

Scalable NIO Servers – Performance

As a continuation of evaluating an NIO server for my iPhone game, I started with looking at pure performance. First, the following links provide already existant benchmarks:

I took my own samples and quickly ran some basic tests as well to just get a really rough idea.  Note that these metrics should be taken with a large grain of salt as they do not attempt to optimize any of the test libraries and the clients are all run in traditional threads on non-server class machines alongside the actual servers.  Thus, these tests are quickly CPU bound on the client.  The idea is to just get a very rough estimate to compare alongside the above tests.

That point aside, my test basically creates a echo server and echo client.  It then starts up X simultaneous threads that push a simple ‘testing’ string back and forth to the server as fast as it can.  The end result is the time it takes to send a message and receive the echo in milliseconds.

50 Threads

Netty:  ~2.75 ms / transaction

Grizzly:  ~2.52 ms / transaction

Mina:  ~3.45 ms / transaction

xSocket:  ~4.15 ms / transaction

100 Threads

Netty:  ~5.23 ms / transaction

Grizzly:  ~5.51 ms / transaction

Mina:  ~7.19 ms / transaction

xSocket:  ~8.23 ms / transaction

Basically, Netty and Grizzly are very similar in their results.  Mina and xSocket are considerably worse.  As such, we will look into memory usage overall and over time in my next post as we continue to drill down to selecting an ideal library.

Reference: http://www.znetdevelopment.com/blogs/2009/04/07/scalable-nio-servers-part-1-performance/

MongoDB – Geospatial Queries

My Adventures in Coding

Most of the data we work with has geographic longitude/latitude information so we have recently started exploring doing some work with geospatial queries using different database technologies (MySQL, Oracle). We currently use MongoDB as our primary data store and we were surprised to find out that MongoDB already has this functionality built in. After doing some reading we also realized that it handles each of our use cases. Even though we use MongoDB everyday, we just never thought a document store would come with this functionality. For more information go to MongoDB Geospatial Indexing.

Download and Setup MongoDB

For this tutorial, we will just use the MongoDB interactive shell for interacting with the database because it is simple. There is no need to complicate this example by using the MongoDB Scala or Python drivers. They are simple to use once you gain an understanding of MongoDB. The basics are…

View original post 638 more words

Data Modeling 101

Data Modeling 101

The goals of this article are to overview fundamental data modeling skills that all developers should have, skills that can be applied on both traditional projects that take a serial approach to agile projects that take an evolutionary approach. My personal philosophy is that every IT professional should have a basic understanding of data modeling. They don’t need to be experts at data modeling, but they should be prepared to be involved in the creation of such a model, be able to read an existing data model, understand when and when not to create a data model, and appreciate fundamental data design techniques. This article is a brief introduction to these skills.  The primary audience for this article is application developers who need to gain an understanding of some of the critical activities performed by an Agile DBA. This understanding should lead to an appreciation of what Agile DBAs do and why they do them, and it should help to bridge the communication gap between these two roles.

Table of Contents

  1. What is data modeling?
  2. How to model data
  3. Evolutionary/agile data modeling
  4. How to become better at modeling data

 

1. What is Data Modeling?

Data modeling is the act of exploring data-oriented structures. Like other modeling artifacts data models can be used for a variety of purposes, from high-level conceptual models to physical data models.  From the point of view of an object-oriented developer data modeling is conceptually similar to class modeling. With data modeling you identify entity types whereas with class modeling you identify classes. Data attributes are assigned to entity types just as you would assign attributes and operations to classes.  There are associations between entities, similar to the associations between classes – relationships, inheritance, composition, and aggregation are all applicable concepts in data modeling.

Traditional data modeling is different from class modeling because it focuses solely on data – class models allow you to explore both the behavior and data aspects of your domain, with a data model you can only explore data issues.  Because of this focus data modelers have a tendency to be much better at getting the data “right” than object modelers. However, some people will model database methods (stored procedures, stored functions, and triggers) when they are physical data modeling. It depends on the situation of course, but I personally think that this is a good idea and promote the concept in my UML data modeling profile (more on this later).

Although the focus of this article is data modeling, there are often alternatives to data-oriented artifacts (never forget Agile Modeling’s Multiple Models principle). For example, when it comes to conceptual modeling ORM diagrams aren’t your only option – In addition to LDMs it is quite common for people to create UML class diagrams and even Class Responsibility Collaborator (CRC) cards instead.  In fact, my experience is that CRC cards are superior to ORM diagrams because it is very easy to get project stakeholders actively involved in the creation of the model. Instead of a traditional, analyst-led drawing session you can instead facilitate stakeholders through the creation of CRC cards.

 

1.1 How are Data Models Used in Practice?

Although methodology issues are covered later, we need to discuss how data models can be used in practice to better understand them.  You are likely to see three basic styles of data model:

 

  • Conceptual data models. These models, sometimes called domain models, are typically used to explore domain concepts with project stakeholders.  On Agile teams high-level conceptual models are often created as part of your initial requirements envisioning efforts as they are used to explore the high-level static business structures and concepts. On traditional teams conceptual data models are often created as the precursor to LDMs or as alternatives to LDMs.
  • Logical data models (LDMs).  LDMs are used to explore the domain concepts, and their relationships, of your problem domain.  This could be done for the scope of a single project or for your entire enterprise. LDMs depict the logical entity types, typically referred to simply as entity types, the data attributes describing those entities, and the relationships between the entities. LDMs are rarely used on Agile projects although often are on traditional projects (where they rarely seem to add much value in practice).
  • Physical data models (PDMs). PDMs are used to design the internal schema of a database, depicting the data tables, the data columns of those tables, and the relationships between the tables. PDMs often prove to be useful on both Agile and traditional projects and as a result the focus of this article is on physical modeling.

Data and Databases

Although LDMs and PDMs sound very similar, and they in fact are, the level of detail that they model can be significantly different. This is because the goals for each diagram is different – you can use an LDM to explore domain concepts with your stakeholders and the PDM to define your database design. Figure 1 presents a simple LDM and Figure 2 a simple PDM, both modeling the concept of customers and addresses as well as the relationship between them.  Both diagrams apply the Barker notationsummarized below.  Notice how the PDM shows greater detail, including an associative table required to implement the association as well as the keys needed to maintain the relationships. More on these concepts later. PDMs should also reflect your organization’s database naming standards, in this case an abbreviation of the entity name is appended to each column name and an abbreviation for “Number” was consistently introduced.  A PDM should also indicate the data types for the columns, such as integer and char(5). Although Figure 2 does not show them, lookup tables (also called reference tables or description tables) for how the address is used as well as for states and countries are implied by the attributes ADDR_USAGE_CODE,STATE_CODE, and COUNTRY_CODE.

 

 

 

Figure 1. A simple logical data model.

 

Simple LDM

 

 

 

Figure 2. A simple physical data model.

 

Simple PDM

 

 

An important observation about Figures 1 and 2 is that I’m not slavishly following Barker’s approach to naming relationships.  For example, between Customer and Address there really should be two names “Each CUSTOMER may be located in one or more ADDRESSES” and “Each ADDRESS may be the site of one or more CUSTOMERS”.  Although these names explicitly define the relationship I personally think that they’re visual noise that clutter the diagram.  I prefer simple names such as “has” and then trust my readers to interpret the name in each direction. I’ll only add more information where it’s needed, in this case I think that it isn’t. However, a significant advantage of describing the names the way that Barker suggests is that it’s a good test to see if you actually understand the relationship – if you can’t name it then you likely don’t understand it.

 

Data models can be used effectively at both the enterprise level and on projects. Enterprise architects will often create one or more high-level LDMs that depict the data structures that support your enterprise, models typically referred to as enterprise data models or enterprise information models.  An enterprise data model is one of several views that your organization’s enterprise architects may choose to maintain and support – other views may explore your network/hardware infrastructure, your organization structure, your software infrastructure, and your business processes (to name a few).  Enterprise data models provide information that a project team can use both as a set of constraints as well as important insights into the structure of their system.

 

Project teams will typically create LDMs as a primary analysis artifact when their implementation environment is predominantly procedural in nature, for example they are using structured COBOL as an implementation language. LDMs are also a good choice when a project is data-oriented in nature, perhaps a data warehouse or reporting system is being developed (having said that, experience seems to show that usage-centered approaches appear to work even better).  However LDMs are often a poor choice when a project team is using object-oriented or component-based technologies because the developers would rather work with UML diagrams or when the project is not data-oriented in nature. As Agile Modeling advises, apply the right artifact(s) for the job. Or, as your grandfather likely advised you, use the right tool for the job. It’s important to note that traditional approaches to Master Data Management (MDM) will often motivate the creation and maintenance of detailed LDMs, an effort that is rarely justifiable in practice when you consider the total cost of ownership (TCO) when calculating the return on investment (ROI) of those sorts of efforts.

 

When a relational database is used for data storage project teams are best advised to create a PDMs to model its internal schema.  My experience is that a PDM is often one of the critical design artifacts for business application development projects.

 

 

 

2.2. What About Conceptual Models?

Halpin (2001) points out that many data professionals prefer to create an Object-Role Model (ORM), an example is depicted in Figure 3, instead of an LDM for a conceptual model.  The advantage is that the notation is very simple, something your project stakeholders can quickly grasp, although the disadvantage is that the models become large very quickly. ORMs enable you to first explore actual data examples instead of simply jumping to a potentially incorrect abstraction – for example Figure 3 examines the relationship between customers and addresses in detail.   For more information about ORM, visit www.orm.net.

 

 

Figure 3. A simple Object-Role Model.

 

My experience is that people will capture information in the best place that they know. As a result I typically discard ORMs after I’m finished with them.  I sometimes user ORMs to explore the domain with project stakeholders but later replace them with a more traditional artifact such as an LDM, a class diagram, or even a PDM. As a generalizing specialist, someone with one or more specialties who also strives to gain general skills and knowledge, this is an easy decision for me to make; I know that this information that I’ve just “discarded” will be captured in another artifact – a model, the tests, or even the code – that I understand.  A specialist who only understands a limited number of artifacts and therefore “hands-off” their work to other specialists doesn’t have this as an option. Not only are they tempted to keep the artifacts that they create but also to invest even more time to enhance the artifacts. Generalizing specialists are more likely than specialists to travel light. The Object Primer 3rd Edition: Agile Model Driven Development (AMDD) with UML 2

 

2.3. Common Data Modeling Notations

Figure 4 presents a summary of the syntax of four common data modeling notations: Information Engineering (IE), Barker, IDEF1X, and the Unified Modeling Language (UML).  This diagram isn’t meant to be comprehensive, instead its goal is to provide a basic overview.  Furthermore, for the sake of brevity I wasn’t able to depict the highly-detailed approach to relationship naming that Barker suggests. Although I provide a brief description of each notation in Table 1 I highly suggest David Hay’s paper A Comparison of Data Modeling Techniques as he goes into greater detail than I do.

 

 

Figure 4. Comparing the syntax of common data modeling notations.

 

Data modeling notation summary

 

 

 

Table 1. Discussing common data modeling notations.

 

Notation

Comments

IE The IE notation (Finkelstein 1989) is simple and easy to read, and is well suited for high-level logical and enterprise data modeling. The only drawback of this notation, arguably an advantage, is that it does not support the identification of attributes of an entity. The assumption is that the attributes will be modeled with another diagram or simply described in the supporting documentation.
Barker The Barker notation is one of the more popular ones, it is supported by Oracle’s toolset, and is well suited for all types of data models. It’s approach to subtyping can become clunky with hierarchies that go several levels deep.
IDEF1X This notation is overly complex.  It was originally intended for physical modeling but has been misapplied for logical modeling as well. Although popular within some U.S. government agencies, particularly the Department of Defense (DoD), this notation has been all but abandoned by everyone else. Avoid it if you can.
UML This is not an official data modeling notation (yet).  Although several suggestions for a data modeling profile for the UML exist, none are complete and more importantly are not “official” UML yet. However, the Object Management Group (OMG) in December 2005 announced an RFP for data-oriented models.

 

 

3. How to Model Data

It is critical for an application developer to have a grasp of the fundamentals of data modeling so they can not only read data models but also work effectively with Agile DBAs who are responsible for the data-oriented aspects of your project. Your goal reading this section is not to learn how to become a data modeler, instead it is simply to gain an appreciation of what is involved.

 

The following tasks are performed in an iterative manner: 

 

The Data Modeling Handbook

Very good practical books about data modeling include Joe Celko’s Data & Databases and Data Modeling for Information Professionals as they both focus on practical issues with data modeling.  The Data Modeling Handbook and Data Model Patterns are both excellent resources once you’ve mastered the fundamentals.  An Introduction to Database Systems is a good academic treatise for anyone wishing to become a data specialist.

 

 

3.1 Identify Entity Types

An entity type, also simply called entity (not exactly accurate terminology, but very common in practice), is similar conceptually to object-orientation’s concept of a class – an entity type represents a collection of similar objects.  An entity type could represent a collection of people, places, things, events, or concepts. Examples of entities in an order entry system would include CustomerAddressOrderItem, and Tax. If you were class modeling you would expect to discover classes with the exact same names. However, the difference between a class and an entity type is that classes have both data and behavior whereas entity types just have data.

 

Ideally an entity should be normal, the data modeling world’s version of cohesive. A normal entity depicts one concept, just like a cohesive class models one concept. For example, customer and order are clearly two different concepts; therefore it makes sense to model them as separate entities.

 

 

 

3.2 Identify Attributes

Each entity type will have one or more data attributes.  For example, in Figure 1 you saw that the Customer entity has attributes such as First Name and Surname and in Figure 2 that the TCUSTOMER table had corresponding data columns CUST_FIRST_NAME and CUST_SURNAME (a column is the implementation of a data attribute within a relational database).

 

Attributes should also be cohesive from the point of view of your domain, something that is often a judgment call. – in Figure 1 we decided that we wanted to model the fact that people had both first and last names instead of just a name (e.g. “Scott” and “Ambler” vs. “Scott Ambler”) whereas we did not distinguish between the sections of an American zip code (e.g. 90210-1234-5678). Getting the level of detail right can have a significant impact on your development and maintenance efforts. Refactoring a single data column into several columns can be difficult, database refactoring is described in detail in Database Refactoring, although over-specifying an attribute (e.g. having three attributes for zip code when you only needed one) can result in overbuilding your system and hence you incur greater development and maintenance costs than you actually needed.

 

 

 

3.3 Apply Data Naming Conventions

Your organization should have standards and guidelines applicable to data modeling, something you should be able to obtain from your enterprise administrators (if they don’t exist you should lobby to have some put in place). These guidelines should include naming conventions for both logical and physical modeling, the logical naming conventions should be focused on human readability whereas the physical naming conventions will reflect technical considerations.  You can clearly see that different naming conventions were applied in Figures 1 and 2.

 

As you saw in Introduction to Agile Modeling, AM includes the Apply Modeling Standards practice. The basic idea is that developers should agree to and follow a common set of modeling standards on a software project. Just like there is value in following common coding conventions, clean code that follows your chosen coding guidelines is easier to understand and evolve than code that doesn’t, there is similar value in following common modeling conventions.

 

 

 

3.4 Identify Relationships

In the real world entities have relationships with other entities.  For example, customers PLACE orders, customers LIVE AT addresses, and line items ARE PART OF orders. Place, live at, and are part of are all terms that define relationships between entities.  The relationships between entities are conceptually identical to the relationships (associations) between objects.

 

Figure 5 depicts a partial LDM for an online ordering system.  The first thing to notice is the various styles applied to relationship names and roles – different relationships require different approaches.  For example the relationship between Customer and Order has two names, places and is placed by, whereas the relationship between Customer and Address has one.  In this example having a second name on the relationship, the idea being that you want to specify how to read the relationship in each direction, is redundant – you’re better off to find a clear wording for a single relationship name, decreasing the clutter on your diagram.  Similarly you will often find that by specifying the roles that an entity plays in a relationship will often negate the need to give the relationship a name (although some CASE tools may inadvertently force you to do this).  For example the role of billing address and the label billed to are clearly redundant, you really only need one.  For example the role part of that Line Item has in its relationship with Order is sufficiently obvious without a relationship name.

 

Figure 5. A logical data model (Information Engineering notation).

 

Identifying relationships

 

 

You also need to identify the cardinality and optionality of a relationship (the UML combines the concepts of optionality and cardinality into the single concept of multiplicity). Cardinality represents the concept of “how many” whereas optionality represents the concept of “whether you must have something.” For example, it is not enough to know that customers place orders.  How many orders can a customer place?  None, one, or several? Furthermore, relationships are two-way streets: not only do customers place orders, but orders are placed by customers.  This leads to questions like: how many customers can be enrolled in any given order and is it possible to have an order with no customer involved? Figure 5 shows that customers place zero or more orders and that any given order is placed by one customer and one customer only.  It also shows that a customer lives at one or more addresses and that any given address has zero or more customers living at it.

 

Although the UML distinguishes between different types of relationships – associations, inheritance, aggregation, composition, and dependency – data modelers often aren’t as concerned with this issue as much as object modelers are. Subtyping, one application of inheritance, is often found in data models, an example of which is the is a relationship between Item and it’s two “sub entities” Service and Product.   Aggregation and composition are much less common and typically must be implied from the data model, as you see with the part of role that Line Item takes with Order. UML dependencies are typically a software construct and therefore wouldn’t appear on a data model, unless of course it was a very highly detailed physical model that showed how views, triggers, or stored procedures depended on other aspects of the database schema.

 

 

 

3.5 Apply Data Model Patterns

Some data modelers will apply common data model patterns, David Hay’s book Data Model Patterns is the best reference on the subject, just as object-oriented developers will apply analysis patterns (Fowler 1997Ambler 1997) and design patterns (Gamma et al. 1995).  Data model patterns are conceptually closest to analysis patterns because they describe solutions to common domain issues.  Hay’s book is a very good reference for anyone involved in analysis-level modeling, even when you’re taking an object approach instead of a data approach because his patterns model business structures from a wide variety of business domains.

Data Model Patterns

 

3.6 Assign Keys

There are two fundamental strategies for assigning keys to tables.  First, you could assign a natural key which is one or more existing data attributes that are unique to the business concept.  The Customer table of Figure 6 there was two candidate keys, in this case CustomerNumber and SocialSecurityNumber. Second, you could introduce a new column, called a surrogate key, which is a key that has no business meaning. An example of which is the AddressID column of the Address table in Figure 6. Addresses don’t have an “easy” natural key because you would need to use all of the columns of the Address table to form a key for itself (you might be able to get away with just the combination of Street and ZipCode depending on your problem domain), therefore introducing a surrogate key is a much better option in this case.

 

Figure 6. Customer and Address revisited (UML notation).

Let’s consider Figure 6 in more detail. Figure 6 presents an alternative design to that presented in Figure 2, a different naming convention was adopted and the model itself is more extensive. In Figure 6 the Customer table has the CustomerNumber column as its primary key and SocialSecurityNumber as an alternate key. This indicates that the preferred way to access customer information is through the value of a person’s customer number although your software can get at the same information if it has the person’s social security number.  The CustomerHasAddress table has a composite primary key, the combination ofCustomerNumber and AddressID.  A foreign key is one or more attributes in an entity type that represents a key, either primary or secondary, in another entity type.  Foreign keys are used to maintain relationships between rows.  For example, the relationships between rows in the CustomerHasAddress table and the Customer table is maintained by the CustomerNumber column within the CustomerHasAddress table. The interesting thing about the CustomerNumber column is the fact that it is part of the primary key for CustomerHasAddress as well as the foreign key to the Customer table. Similarly, the AddressID column is part of the primary key of CustomerHasAddress as well as a foreign key to the Address table to maintain the relationship with rows of Address.

 

Although the “natural vs. surrogate” debate is one of the great religious issues within the data community, the fact is that neither strategy is perfect and you’ll discover that in practice (as we see in Figure 6 ) sometimes it makes sense to use natural keys and sometimes it makes sense to use surrogate keys. In Choosing a Primary Key: Natural or Surrogate? I describe the relevant issues in detail.

 

 

3.7 Normalize to Reduce Data Redundancy

Data normalization is a process in which data attributes within a data model are organized to increase the cohesion of entity types.  In other words, the goal of data normalization is to reduce and even eliminate data redundancy, an important consideration for application developers because it is incredibly difficult to stores objects in a relational database that maintains the same information in several places.  Table 2 summarizes the three most common normalization rules describing how to put entity types into a series of increasing levels of normalization. Higher levels of data normalization (Date 2000) are beyond the scope of this book.  With respect to terminology, a data schema is considered to be at the level of normalization of its least normalized entity type.  For example, if all of your entity types are at second normal form (2NF) or higher then we say that your data schema is at 2NF.

Data Modeling for Information Professionals

Table 2. Data Normalization Rules.

 

Level Rule
First normal form (1NF) An entity type is in 1NF when it contains no repeating groups of data.
Second normal form (2NF) An entity type is in 2NF when it is in 1NF and when all of its non-key attributes are fully dependent on its primary key.
Third normal form (3NF) An entity type is in 3NF when it is in 2NF and when all of its attributes are directly dependent on the primary key.

 

 

Figure 7 depicts a database schema in ONF whereas Figure 8 depicts a normalized schema in 3NF. Read the Introduction to Data Normalization essay for details.

 

Why data normalization?  The advantage of having a highly normalized data schema is that information is stored in one place and one place only, reducing the possibility of inconsistent data. Furthermore, highly-normalized data schemas in general are closer conceptually to object-oriented schemas because the object-oriented goals of promoting high cohesion and loose coupling between classes results in similar solutions (at least from a data point of view). This generally makes it easier to map your objects to your data schema.  Unfortunately, normalization usually comes at a performance cost.  With the data schema ofFigure 7 all the data for a single order is stored in one row (assuming orders of up to nine order items), making it very easy to access.  With the data schema of Figure 7 you could quickly determine the total amount of an order by reading the single row from the Order0NF table.  To do so with the data schema of Figure 8 you would need to read data from a row in the Order table, data from all the rows from the OrderItem table for that order and data from the corresponding rows in the Item table for each order item. For this query, the data schema of Figure 7 very likely provides better performance.

 

 

 

Figure 7. An Initial Data Schema for Order (UML Notation).

 

Order in 0NF

 

 

 

Figure 8. A normalized schema in 3NF (UML Notation).

 

Order Fully Normalized

 

 

In class modeling, there is a similar concept called Class Normalization although that is beyond the scope of this article.

3.8 Denormalize to Improve Performance

Normalized data schemas, when put into production, often suffer from performance problems. This makes sense – the rules of data normalization focus on reducing data redundancy, not on improving performance of data access.  An important part of data modeling is to denormalize portions of your data schema to improve database access times.  For example, the data model of Figure 9 looks nothing like the normalized schema of Figure 8. To understand why the differences between the schemas exist you must consider the performance needs of the application. The primary goal of this system is to process new orders from online customers as quickly as possible. To do this customers need to be able to search for items and add them to their order quickly, remove items from their order if need be, then have their final order totaled and recorded quickly. The secondary goal of the system is to the process, ship, and bill the orders afterwards.

 

 

 

Figure 9. A Denormalized Order Data Schema (UML notation).

 

Order Denormalized

 

 

 

To denormalize the data schema the following decisions were made:

 

  1. To support quick searching of item information the Item table was left alone.
  2. To support the addition and removal of order items to an order the concept of an OrderItem table was kept, albeit split in two to support outstanding orders and fulfilled orders. New order items can easily be inserted into the OutstandingOrderItem table, or removed from it, as needed.
  3. To support order processing the Order and OrderItem tables were reworked into pairs to handle outstanding and fulfilled orders respectively. Basic order information is first stored in the OutstandingOrder and OutstandingOrderItem tables and then when the order has been shipped and paid for the data is then removed from those tables and copied into the FulfilledOrder and FulfilledOrderItem tables respectively. Data access time to the two tables for outstanding orders is reduced because only the active orders are being stored there. On average an order may be outstanding for a couple of days, whereas for financial reporting reasons may be stored in the fulfilled order tables for several years until archived. There is a performance penalty under this scheme because of the need to delete outstanding orders and then resave them as fulfilled orders, clearly something that would need to be processed as a transaction.
  4. The contact information for the person(s) the order is being shipped and billed to was also denormalized back into the Order table, reducing the time it takes to write an order to the database because there is now one write instead of two or three.  The retrieval and deletion times for that data would also be similarly improved.

Note that if your initial, normalized data design meets the performance needs of your application then it is fine as is.  Denormalization should be resorted to only when performance testing shows that you have a problem with your objects and subsequent profiling reveals that you need to improve database access time. As my grandfather said, if it ain’t broke don’t fix it.

 

 

5. Evolutionary/Agile Data Modeling

Evolutionary data modeling is data modeling performed in an iterative and incremental manner. The article Evolutionary Development explores evolutionary software development in greater detail.  Agile data modeling is evolutionary data modeling done in a collaborative manner.  The article Agile Data Modeling: From Domain Modeling to Physical Modeling works through a case study which shows how to take an agile approach to data modeling.

Although you wouldn’t think it, data modeling can be one of the most challenging tasks that an Agile DBA can be involved with on an agile software development project. Your approach to data modeling will often be at the center of any controversy between the agile software developers and the traditional data professionals within your organization. Agile software developers will lean towards an evolutionary approach where data modeling is just one of many activities whereas traditional data professionals will often lean towards a big design up front (BDUF) approach where data models are the primary artifacts, if not THE artifacts.  This problem results from a combination of the cultural impedance mismatch, a misguided need to enforce the “one truth”, and “normal” political maneuvering within your organization. As a result Agile DBAs often find that navigating the political waters is an important part of their data modeling efforts.

 

6. How to Become Better At Modeling Data

How do you improve your data modeling skills?  Practice, practice, practice. Whenever you get a chance you should work closely with Agile DBAs, volunteer to model data with them, and ask them questions as the work progresses.  Agile DBAs will be following the AM practice Model With Others so should welcome the assistance as well as the questions – one of the best ways to really learn your craft is to have someone as “why are you doing it that way”. You should be able to learn physical data modeling skills from Agile DBAs, and often logical data modeling skills as well.

 

Similarly you should take the opportunity to work with the enterprise architects within your organization.  As you saw in Agile Enterprise Architecture they should be taking an active role on your project, mentoring your project team in the enterprise architecture (if any), mentoring you in modeling and architectural skills, and aiding in your team’s modeling and development efforts. Once again, volunteer to work with them and ask questions when you are doing so.  Enterprise architects will be able to teach you conceptual and logical data modeling skills as well as instill an appreciation for enterprise issues.

 

You also need to do some reading.  Although this article is a good start it is only a brief introduction.  The best approach is to simply ask the Agile DBAs that you work with what they think you should read.

 

My final word of advice is that it is critical for application developers to understand and appreciate the fundamentals of data modeling.  This is a valuable skill to have and has been since the 1970s.  It also provides a common framework within which you can work with Agile DBAs, and may even prove to be the initial skill that enables you to make a career transition into becoming a full-fledged Agile DBA. –

Reference: http://www.agiledata.org/essays/dataModeling101.html

Your action plan: using feedback to drive your career

Your action plan: using feedback to drive your career

Once you’ve sought feedback, it’s time to make choices about what you’re going to do, share the plan with your manager, and gauge your progress as you work on it.

The negatives

If you’ve asked for feedback, listened, and recorded it, you’re ready to start creating an action plan. You now need to decide on the importance of each constructive piece of feedback. Here are some things to consider:

  • How many times did you hear it? The more times, the more important
  • Who did you hear it from? Worry more about your boss’s opinion than your peers’, and more about your peers’ than anyone else’s
  • When did they say it? The second thing is often the most important – people warm up with a gentle message, and often end later with low priority points
  • Is it a perception or a reality? Did you have an off-day, or an off-interaction that was out of character? Or is this a genuine flaw?
  • Do you think it is correct? Was it on your list?

I recommend finding the top five pieces of feedback, and sorting them from most- to least-important by considering the criteria above. You don’t want to work on too many things at once.

The positives

Don’t take positive feedback for granted. You can use the same techniques as you’ve used for the negatives to create your list of top strengths. Make sure you do this too: figure out your top five strengths.

Creating a Plan

You have a fundamental choice in creating a plan. You can decide to lean hard on your strengths and have them propel you further forward, or you can choose to work on the weaknesses so they don’t hold you back. The right thing to do is usually somewhere in between: focus on improving a couple of weaknesses at any one time, and work on using your strengths to their maximum potential.

One thing I’ve observed is that senior people are usually held back by their weaknesses. In part it’s the Peter Principle, and in part it’s the fact that everyone around them is pretty awesome, and flaws stand out. There’s definitely a point I often see where people get confused – they expect to advance  in their career because of the awesome competencies they have, and then suddenly they’re stuck because of their weaknesses. It often takes people a while to accept that a few things need to change, especially if they’ve never heard negative feedback before.

I’d recommend taking your top two or three negatives, and your top one or two strengths, and writing them down as the areas you want to focus on. Bonus points if you sort them into priority order. Now, you need an action plan. Next to each point, write key steps you’ll take to address that weakness, or showcase that strength. The more actionable, the better – it’s not that helpful to say “you’ll improve your public speaking”, it’s helpful to say “give three public talks in 2012, and seek actionable feedback immediately after each presentation”.  Try and use quantities, dates, names, situations, or other concrete points in creating the plan. Remember that taking action on weaknesses takes you out of your comfort zone – so the steps should feel hard, awkward, and uncertain.

Executing the plan

If you’ve got an actionable plan, I’d recommend reviewing it with your boss. At the very least, you’re going to look good for having taken career development seriously. You’ll probably also get great feedback on whether this is a good plan or not – and, again, your boss’s opinion matters.

Now you can go execute your plan. Good luck. Keep an eye on your progress: I’d recommend giving yourself a green, yellow, or red rating on each point every six weeks or so. Ask the people who gave you’re the feedback points whether they’re seeing a change – the door is open with them, and you should use it.

Of course, this is a process that never ends. You can complete the plan successfully, and there’ll be another plan ready to be created by starting over. Good luck, I hope you use feedback successfully in building your career.

Reference: http://hughewilliams.com/2012/08/

Don’t use a Pie Chart

Don’t use a Pie Chart

Pie charts in action.

Take the example above. Take a look at Pie Chart A. The blue slice looks the same size as the red or green slice. You might draw the conclusion they’re roughly the same. In fact, take a look at the histogram below — the red is 17 units, blue is 18 units, and green is 20 units. The histogram is informative, useful for comparison, and clear for communication. (There’s still a cardinal sin here though: no labels on the axes; I’ll rant about that some other day.)

Compare pie charts B and C above. It sure looks like there’s the same quantity of blue in B and C, and about the same amount of green. The quantities aren’t the same: the histograms below show that green is 19 and 20 respectively, and blue is 20 and 22.

You might argue that pie charts are useful for comparing relative quantities. I’d argue it’s possible but difficult to interpret. Take a look at three yellow slices in charts A, B, and C. They look to be similar percentages of the pies, but it’s hard to tell: they are oriented slightly differently, making the comparison unclear. What’s the alternative? Use a histogram with the y-axis representing the percentage.

Pie charts get even more ugly when there’s lots of slices, or some of the slices are ridiculously small. If you’re going to use them — I still don’t think it’s a good idea — then use them when there are only a few values and the smallest slices can be labeled. The pie chart below is ridiculous.

A pie chart that's nearly impossible to read. I'm not sure what conclusions you could draw from this one.

Reference: http://hughewilliams.com/2014/02/13/dont-use-a-pie-chart/