Skip to content

OctaveDouble precision differences due to octave hex representation  #3

Description

@karlulricsimon

In testIoOctaveDouble.java I added the following test

    @Test
    public void testPrecision() {
    	final OctaveEngine octave = new OctaveEngineFactory().getScriptEngine();
    	octave.eval("format long");
    	octave.eval("a = (1/3) * 0.29");
    	OctaveDouble a = octave.get(OctaveDouble.class, "a");
    	//This assertion will fail.
    	//On evaluation, the console will report "a =  0.0966666666666667",
    	//But retrieving the value back from the wrapper reports the native-hex value cast to a double
    	assertEquals(0.096666666666667, a.get(1), 0.0);
    }

Within the command window of octave, the value of the variable a will be "0.0966666666666667", but if the format is changed to 'native-hex' the value of a is 'be58f28b25bfb83f' which converts to "0.09666666666666665"

It seems that this is a bug in how the wrapper stores the variable value, though it could be that my expectations are incorrect.

I'm using Ubuntu 16.04
octaveInJava version 0.6.8
Octave version 4.3.0

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions