Advanced Windows PowerShell Scripting Video Training

Advanced Windows PowerShell Scripting Video Training
Advanced Windows PowerShell Scripting Video Training

Tuesday, May 12, 2015

Accuracy problem with System.Double and System.Single

So, what do I do while my class is working on labs?  Well, if I’m not answer emails from the Navy, I’m working on some side projects.  Right now, I’m working on a project to grab meta data via geographical coordinates. When I develop code, I brake the objective down into manageable parts.  I then look at each part and ask myself if I know how to do it.  For each part that I do not know how, I start writing code to discover the path to success.

Well, while working on a bit of code, I needed to increment my value for latitude by .05.  Something odd happened so I wrote out a small bit of code to focus in on the issue.  Here is the code.

$Increment = .05

$Start = 0

$Value2 = $Start

 

While ($Value2 -lt 10)

{

    Write-host $Value2

    $Value2 += $Increment

}

The data type of $Value2 is Double.  Everything starts out fine.

0

0.05

0.1

0.15

0.2

Until…

3.45

3.5

3.55

3.6

3.65

3.69999999999999

3.74999999999999

3.79999999999999

3.84999999999999

 

Why System.Double start incrementing in .049999999999999 and then back to .05 later on, I do not know.  OK, let’s force the variable to be System.Single

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5000001

0.5500001

0.6000001

0.6500001

 

OK, now it increments by 0.050001 and then back to .05. 

Luckily, I found a data type that does not loose it’s accuracy under these conditions, System.Decimal.

$Increment = .05

$Start = 0

[Decimal]$Value4 = $Start

 

While ($Value4 -lt 10)

{

    Write-host $Value4

    $Value4 += $Increment

 

 

}

$Value4 | GM

 

Below is a sample of the desired output.

0

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

 

I took a look at System.Double on MSDN. According to Microsoft, System.Double coheres to the standard for binary floating-point arithmetic. (See the Remarks section).  Since some fractional values cannot be represented precisely.  Look at System.Decimal.  In the Remarks section, you will see that it dose not have a round off error.  At least not at the precision level that I need.

I now need to decide between the smaller memory footprint of Single or Double data types as compared to the much larger storage requirement of Decimal.  It looks like that I will be computing the Longitude and Latitude values in Decimal, But using the –as operator in PowerShell to convert them into Single for storage since I do not need to go beyond 2 decimal places.

[Decimal]$Decimal = 5.5

$Single = $Decimal -as [Single]

$Single

 

The result will be a Single value that will have the precision that I need with a smaller memory footprint.

No comments: