Improve write speed

[Krons]

Hi!

I have the following code to write in a file some info. The problem is this method is too slow when it's try to write a 8mb file. ¿What can I do to improve the speed?. It takes more than 1 min now...Thanks!

    public Boolean guardarJugadores(TreeMap hashJugadores){

        Boolean res = false;

        try {

            RandomAccessFile fichero = new RandomAccessFile(hashOpciones.get("jugadores").toString()+".temp", "rw");
            Iterator it = hashJugadores.values().iterator();
            Iterator itRegistros = hashJugadores.values().iterator();

            fichero.writeDouble(Constantes.principioArchivo);
            fichero.writeDouble(0);
            fichero.writeInt(Integer.reverseBytes(Constantes.cantJugadores));

            while(it.hasNext()){

                Jugador jug = (Jugador) it.next();
                Cabecera cab = jug.getCabecera();

                fichero.writeInt(Integer.reverseBytes(cab.getId()));
                fichero.writeByte(0);
                fichero.writeInt(Integer.reverseBytes(cab.getInicio()));
                fichero.writeInt(Integer.reverseBytes(cab.getLargo()));

                jug = null;
                cab = null;
                
            }

            while(itRegistros.hasNext()){

                Jugador jugador = (Jugador) itRegistros.next();

                fichero.writeShort(Short.reverseBytes(jugador.getSumaLargos()));
                fichero.writeShort(Short.reverseBytes(Constantes.inicioRegistro));
                fichero.writeBoolean(jugador.getTipoJugador());
                cFuncion.writeUnsignedShort(jugador.getIdJugador(), fichero);
                fichero.write(jugador.getDorsal());

                fichero.writeShort(Short.reverseBytes(jugador.getLargoNombreCorto()));
                cFuncion.grabarCadena(jugador.getNombreCorto(), jugador.getLargoNombreCorto(), fichero);
                
                fichero.writeShort(Short.reverseBytes(jugador.getLargoNombre()));
                cFuncion.grabarCadena(jugador.getNombre(), jugador.getLargoNombre(), fichero);

                fichero.write(jugador.getDorsalGenerico());
                fichero.writeBoolean(jugador.getUtilizaDorsalGenerico());

                fichero.write(jugador.getRol1());
                fichero.write(jugador.getRol2());
                fichero.write(jugador.getRol3());
                fichero.write(jugador.getRol4());
                fichero.write(jugador.getRol5());
                fichero.write(jugador.getRol6());

                fichero.write(jugador.getNacionalidad());
                fichero.write(jugador.getColorPiel());
                fichero.write(jugador.getColorPelo());
                fichero.write(jugador.getPosicion());
                fichero.write(jugador.getPelo());
                fichero.write(jugador.getBelloFacial());
                fichero.writeBoolean(jugador.getNacionalizado());

                fichero.write(jugador.getDiaNacimiento());
                fichero.write(jugador.getMesNacimiento());
                fichero.writeShort(Short.reverseBytes(jugador.getAnyoNacimiento()));

                fichero.write(jugador.getAltura());
                fichero.write(jugador.getPeso());

                if(!jugador.getTipoJugador()){
                    fichero.write(jugador.getPaisNatal());

                    fichero.writeShort(Short.reverseBytes(jugador.getLargoCiudadNacimiento()));
                    cFuncion.grabarCadena(jugador.getCiudadNacimiento(), jugador.getLargoCiudadNacimiento(), fichero);

                    fichero.writeShort(Short.reverseBytes(jugador.getLargoEquipoProcedencia()));
                    cFuncion.grabarCadena(jugador.getEquipoProcedencia(), jugador.getLargoEquipoProcedencia(), fichero);

                    fichero.writeShort(Short.reverseBytes(jugador.getLargoVecesInternacional()));
                    cFuncion.grabarCadena(jugador.getVecesInternacional(), jugador.getLargoVecesInternacional(), fichero);

                    fichero.writeShort(Short.reverseBytes(jugador.getLargoCadena1()));
                    cFuncion.grabarCadena(jugador.getCadena1(), jugador.getLargoCadena1(), fichero);

                    fichero.writeShort(Short.reverseBytes(jugador.getLargoCualidadesTecnicas()));
                    cFuncion.grabarCadena(jugador.getCualidadesTecnicas(), jugador.getLargoCualidadesTecnicas(), fichero);

                    fichero.writeShort(Short.reverseBytes(jugador.getLargoPalmares()));
                    cFuncion.grabarCadena(jugador.getPalmares(), jugador.getLargoPalmares(), fichero);

                    fichero.writeShort(Short.reverseBytes(jugador.getLargoInternacional()));
                    cFuncion.grabarCadena(jugador.getInternacional(), jugador.getLargoInternacional(), fichero);

                    fichero.writeShort(Short.reverseBytes(jugador.getLargoAnecdotario()));
                    cFuncion.grabarCadena(jugador.getAnecdotario(), jugador.getLargoAnecdotario(), fichero);

                    fichero.writeShort(Short.reverseBytes(jugador.getLargoUltimaTemporada()));
                    cFuncion.grabarCadena(jugador.getUltimaTemporada(), jugador.getLargoUltimaTemporada(), fichero);

                    fichero.writeShort(Short.reverseBytes(jugador.getLargoTrayectoria()));
                    cFuncion.grabarCadena(jugador.getTrayectoria(), jugador.getLargoTrayectoria(), fichero);
                }

                fichero.write(jugador.getVelocidad());
                fichero.write(jugador.getResistencia());
                fichero.write(jugador.getAgresividad());
                fichero.write(jugador.getCalidad());

                fichero.write(jugador.getRemate());
                fichero.write(jugador.getRegate());
                fichero.write(jugador.getPase());
                fichero.write(jugador.getTiro());
                fichero.write(jugador.getEntradas());
                fichero.write(jugador.getPortero());

                fichero.write(jugador.getPie());
                fichero.write(jugador.getPenalti());
                fichero.write(jugador.getCornerIzquierdo());
                fichero.write(jugador.getCornerDerecho());
                fichero.write(jugador.getFaltaIzquierda());
                fichero.write(jugador.getFaltaDerecha());

            }

            fichero.close();
            File fich1 = new File(hashOpciones.get("jugadores").toString());
            File fich2 = new File(hashOpciones.get("jugadores").toString()+".temp");

            fich1.delete();
            fich2.renameTo(fich1);


        } catch (IOException ex) {
            Logger.getLogger(cJugador.class.getName()).log(Level.SEVERE, null, ex);
        }

        return res;
    }

    public void grabarCadena(String cadena, int largo, RandomAccessFile fichero){

        int[] grabar = convCadenaPCFAscii(cadena);

        for(int i = 0;i<=largo-1;i++){
            try {
                fichero.write(grabar[i]);
            } catch (IOException ex) {
                Logger.getLogger(cFunciones.class.getName()).log(Level.SEVERE, null, ex);
            }
        }
    }

[xcallmejudasx]

This is a little past my scope but the only thing I can think of would be adding an extra thread or two to speed up the writing.

I'm assuming you've looked into the different types of trees(AVL, Red Black, etc) and used the fastest one for your needs?

[Krons]

yes, i need to use this kind of tree...

I don't work before with threads, so i'm gonna need to learn it xD.

Using DataOutputStrem will improve the speed?

[xcallmejudasx]

I honestly have no clue. I've never worked with threads or have a familiarity with the DataOutputStream but it seems to make sense that a stream designed for writing data would work faster then a bunch of different write methods. Perhaps one of the Gurus could hop in and give you more useful advice.

[markw8500]

I just started to work with Threads and they might not work out for you... The reason being that from your code it looks like all the data that you are writing to a file needs to be in order and if you mix Threads in with that, then each Thread will be writing to the file different information at different times... So you will probably be overwriting you data and mixing it all up...

[OrangeDog]

A few things:

Don't use a RandomAccessFile, opening one for output only will be quicker.
Use generics for your collections - so a Hash<Jugador> - to avoid casts.
Using an enhanced for loop - for (Jugador j : Jugadores) - will give most efficient iterator for the system.
Restructure the Jugador class so there is, for example, a single write(java.io.OutputStream) method that writes directly from the fields. This will avoid all the repeated method calls for every field.

These should speed up the program to varying extents. If it's still not fast enough try running it through a profiler and seeing where the bottlenecks are. Typically, using the simplest objects (a raw OutputStream) will provide faster methods, but at an ease-of-use cost.

[neilcoffey]

The main thing that determines the speed of I/O is the number of operating system calls you need to make per byte written. The classes BufferedInputStream and BufferedOutputStream (and other cousins such as BufferedReader) aim to minimise the number of I/O calls by storing your data into a buffer, and only actually writing it block at a time. At around 8K/call, you reach a point where there's not much benefit in buffering any more.

The problem with RandomAccessFile is that it doesn't do any buffering. Every time you make a read/write call on RAF, you are making an operating system call. So for example, your grabarCadena() method is making a call for every character written! This is clearly bad.

The solution is to change your code to write via a buffered output stream.

For more information on buffering, see my article on How big should my input stream buffer be? (I believe the figures are essentially similar for output).

The absolute fastest way to write the data is probably to use a mapped ByteBuffer with the NIO package, but this introduces some other problems, so probably not recommended.

[OrangeDog]

A rough order for how much different things slow the program down, slowest first:

Native method calls
Disk operations
Other operating system calls
Java method calls
Memory operations
Java primitive operations (thread-related stuff is slowest)

Obviously, doing quick stuff repeatedly is going to be slower than doing slow stuff once.

[neilcoffey]

OK, bear in mind with the last three:
- the JIT compiler tries to take steps to make sure that method calls per se have close to zero cost when it matters (e.g. if you sit in a loop calling a short-ish method, then the method gets inlined)
- not sure exactly what you mean by "Java primitive operations", but if you mean simply operations on a local variable, then that's probably true, except that the boundary between when you think Java is operating on a "primitive" and on something else isn't necessarily where you think it is. So you might logically expect to be writing to a register/local variable instead of to "main memory", but that ends up not being the case, and vice versa.

[OrangeDog]

-If you sit in a loop making lots of calls including recursion there's not much JIT can do for you. In general, assume maximal uselessness from a compiler and that every method call is going to involve a new stack frame (I know for a fact that Sun's javac doesn't optimise tail recursion)
-Primitive operations are those that have equivalent ops in Java Bytecode. While results differ depending on platform and implementation, you can generally assume that they will be pretty quick, though those that don't compile simply to base machine code may take longer (like object and thread management stuff)

Anyway, this is all rather advanced. Just cut out all those method calls and use a buffered stream should get all the speed-up you need.

[toadaly]

Don't use a RandomAccessFile, opening one for output only will be quicker.

Yep...that'll help. Even more importantly, use a BufferedOutputStream. You want to avoid the transaction costs of the write method.

Substituting the following for the RandomAccessFile will probably make the other optimization choices moot:

BufferedOutputStream fichero = new BufferedOutputStream(new FileOutputStream(hashOpciones.get("jugadores").toString()+".temp"));

(there is no writeShort method for an OutputStream, but it looks like maybe you're starting with bytes anyway?)