sparse command and double vs single arrays

sparse command and double vs single arrays

Post by Henry Wolkowi » Wed, 23 Oct 2002 12:53:31



I am using the command

A=sparse(I,J,values,m,n)

where the index sets I and J are very 'long' vectors.
Therefore, I would like to use
      single arrays   for I and J
to save memory. However, I then get the error message

??? Error using ==> sparse
Function 'sparse' not defined for variables of class 'single'.

I have tried

A=sparse(double(I),double(J),values,m,n)

but this seems to still run into memory problems.
Is there a way to avoid this?

    thanks for any hints/advice
--

Univ. of Waterloo        |URL http://orion.math.uwaterloo.ca/~hwolkowi
Dept of Comb and Opt     |Tel   (519) 888-4567, x5589, office MC6065
Waterloo, Ont. CANADA N2L 3G1          |Fax   (519) 725-5441

 
 
 

sparse command and double vs single arrays

Post by Penny Anderso » Wed, 23 Oct 2002 23:06:40


Henry,

If you can't store double precision i, j and values in the MATLAB
workspace with storage requirements:

3 * 8 * nnz(A)

then you also can't store 32 bit integer/single precision i and j and
double precision values AND the resulting sparse(A) with combined
storage requirements:

(2 * 4 + 8) * nnz(A)  +  (4+8)*nnz(A) + 4*(size(A,2)+1)

However, we are considering accepting integer or single precision i
and j sparse matrix indices in the sparse function S =
sparse(i,j,vals) in a future release of MATLAB. (Internally the row
indices and column pointers of the sparse matrix data structure are
stored as 32 bit integers already).

Penny Anderson
The MathWorks, Inc.


> I am using the command

> A=sparse(I,J,values,m,n)

> where the index sets I and J are very 'long' vectors.
> Therefore, I would like to use
>       single arrays   for I and J
> to save memory. However, I then get the error message

> ??? Error using ==> sparse
> Function 'sparse' not defined for variables of class 'single'.

> I have tried

> A=sparse(double(I),double(J),values,m,n)

> but this seems to still run into memory problems.
> Is there a way to avoid this?

>     thanks for any hints/advice
> --

> Univ. of Waterloo        |URL

http://orion.math.uwaterloo.ca/~hwolkowi

- Show quoted text -

Quote:> Dept of Comb and Opt     |Tel   (519) 888-4567, x5589, office MC6065
> Waterloo, Ont. CANADA N2L 3G1          |Fax   (519) 725-5441


 
 
 

sparse command and double vs single arrays

Post by Henry Wolkowi » Wed, 23 Oct 2002 23:18:31


Thanks for your help - and info. It is good to hear that in the future the
sparse command will accept integer values rather than only double precision.

I have managed to improve the code I am working on but still have one major
bottleneck. Even though I have to perform operations such as a
Cholesky factorization and many matrix multiplications, my bottleneck is
with the following two lines

 mat11=sparse(double(Ix), double(Jx), X(valuex),n^2,n*(n-1)/2);
  mat11=mat11+sparse(double(Is), double(Js), Sy(values),n^2,n*(n-1)/2);

Here Ix,Jx,Is,Js,valuex,values are long vectors of indices and
X,Sy are sparse symmetric matrices n by n matrices.

Have I reached the limit in improving things within matlab? Should I be
looking at a mex file?

      thanks for the advice



>Henry,

>If you can't store double precision i, j and values in the MATLAB
>workspace with storage requirements:

>3 * 8 * nnz(A)

>then you also can't store 32 bit integer/single precision i and j and
>double precision values AND the resulting sparse(A) with combined
>storage requirements:

>(2 * 4 + 8) * nnz(A)  +  (4+8)*nnz(A) + 4*(size(A,2)+1)

>However, we are considering accepting integer or single precision i
>and j sparse matrix indices in the sparse function S =
>sparse(i,j,vals) in a future release of MATLAB. (Internally the row
>indices and column pointers of the sparse matrix data structure are
>stored as 32 bit integers already).

>Penny Anderson
>The MathWorks, Inc.



>> I am using the command

>> A=sparse(I,J,values,m,n)

>> where the index sets I and J are very 'long' vectors.
>> Therefore, I would like to use
>>       single arrays   for I and J
>> to save memory. However, I then get the error message

>> ??? Error using ==> sparse
>> Function 'sparse' not defined for variables of class 'single'.

>> I have tried

>> A=sparse(double(I),double(J),values,m,n)

>> but this seems to still run into memory problems.
>> Is there a way to avoid this?

>>     thanks for any hints/advice
>> --

>> Univ. of Waterloo        |URL
>http://orion.math.uwaterloo.ca/~hwolkowi
>> Dept of Comb and Opt     |Tel   (519) 888-4567, x5589, office MC6065
>> Waterloo, Ont. CANADA N2L 3G1          |Fax   (519) 725-5441

--

Univ. of Waterloo        |URL http://orion.math.uwaterloo.ca/~hwolkowi
Dept of Comb and Opt     |Tel   (519) 888-4567, x5589, office MC6065
Waterloo, Ont. CANADA N2L 3G1          |Fax   (519) 725-5441
 
 
 

sparse command and double vs single arrays

Post by Penny Anderso » Wed, 23 Oct 2002 23:58:33


Henry,

Can you not remove your addition in favour of one sparse call?
Repeasted indices i and j in S = sparse(i,j,s) just add the associated
values in s so you will get the same effect. If the columns of Jx are
all before the columns of Js, then you are in ok shape with your
current approach, but otherwise:

mat11 = sparse([double(Ix); double(Is)], [double(Jx); double(Js)],
[X(valuex); Sy(values)], n^2, n*(n-1)/2);

is still your best bet, especially if [double(Jx); double(Js)] is
sorted as I said before.

Going to a MEX-file for construction will be a little tedious since
the column indices are not stored explicitly - you have to create
column pointers into the array of row indices instead. I'm not sure it
will save you much over a sorted j vector.

Penny Anderson
The MathWorks, Inc.


Quote:> Thanks for your help - and info. It is good to hear that in the
future the
> sparse command will accept integer values rather than only double
precision.

> I have managed to improve the code I am working on but still have
one major
> bottleneck. Even though I have to perform operations such as a
> Cholesky factorization and many matrix multiplications, my
bottleneck is
> with the following two lines

>  mat11=sparse(double(Ix), double(Jx), X(valuex),n^2,n*(n-1)/2);
>   mat11=mat11+sparse(double(Is), double(Js),

Sy(values),n^2,n*(n-1)/2);

> Here Ix,Jx,Is,Js,valuex,values are long vectors of indices and
> X,Sy are sparse symmetric matrices n by n matrices.

> Have I reached the limit in improving things within matlab? Should I
be
> looking at a mex file?

>       thanks for the advice



> >Henry,

> >If you can't store double precision i, j and values in the MATLAB
> >workspace with storage requirements:

> >3 * 8 * nnz(A)

> >then you also can't store 32 bit integer/single precision i and j
and
> >double precision values AND the resulting sparse(A) with combined
> >storage requirements:

> >(2 * 4 + 8) * nnz(A)  +  (4+8)*nnz(A) + 4*(size(A,2)+1)

> >However, we are considering accepting integer or single precision i
> >and j sparse matrix indices in the sparse function S =
> >sparse(i,j,vals) in a future release of MATLAB. (Internally the row
> >indices and column pointers of the sparse matrix data structure are
> >stored as 32 bit integers already).

> >Penny Anderson
> >The MathWorks, Inc.


message

> >> I am using the command

> >> A=sparse(I,J,values,m,n)

> >> where the index sets I and J are very 'long' vectors.
> >> Therefore, I would like to use
> >>       single arrays   for I and J
> >> to save memory. However, I then get the error message

> >> ??? Error using ==> sparse
> >> Function 'sparse' not defined for variables of class 'single'.

> >> I have tried

> >> A=sparse(double(I),double(J),values,m,n)

> >> but this seems to still run into memory problems.
> >> Is there a way to avoid this?

> >>     thanks for any hints/advice
> >> --

> >> Univ. of Waterloo        |URL
> >http://orion.math.uwaterloo.ca/~hwolkowi
> >> Dept of Comb and Opt     |Tel   (519) 888-4567, x5589, office
MC6065
> >> Waterloo, Ont. CANADA N2L 3G1          |Fax   (519) 725-5441

> --

> Univ. of Waterloo        |URL

http://orion.math.uwaterloo.ca/~hwolkowi

- Show quoted text -

Quote:> Dept of Comb and Opt     |Tel   (519) 888-4567, x5589, office MC6065
> Waterloo, Ont. CANADA N2L 3G1          |Fax   (519) 725-5441