Parsing data slow (please help)

Parsing data slow (please help)

Post by Phillip Blo » Thu, 07 Aug 2003 18:55:01

Hi, all

I have written a m-file that parses data from a file. I have tried
numerous approaches and finally I've got something that works. But as
always we must strive for optimum performance. One line in my code
takes up 90+% of the time.

Background: I have a cell array (306307x1). Each cell contains a
string that looks more or less like: ',,,,,,,,,,,,,126,4.53424'. I
need to get to a matrix with strips out the double as vertically
concatenates them. Something like:

a = [0 0 0 0 0 0 0 0 0 0 0 0 0 126 4.5643
     0 0 0 0 0 0 0 0 0 0 0 0 0 234 23.343 ..]

I have used the following:
longString = char(cellOfData{1:end-1})';
longString = [longString;repmat(',',1,size(longString,2))];
longString = longString(:)';

dataMatrix = strread(longString,'%f','delimiter',',')';
dataMatrix = reshape(dataMatrix,15,length(dataMatrix)/15)';
dataMatrix = dataMatrix(:,end-1:end);

The <char> function is taking all the time. Can anyone speed
this up for me? I hope this is enough info.

Thanks in advance


1. Very Slow Training; Please help!


I am I training a feed-forward network
with backpropagation.
There are 15 input neurons
(12 in some cases), 1 hidden layer
and 1 output layer with 1 neuron.

Here is how part of my code looks:

net=newff([0 1; 0 1; 0 1;0 1;0 1;0 1;0 1;0 1;0 1;0 1;0 1;0 1],[12
1],{'tansig' 'purelin'});

net.trainFcn='trainrp'; = 0.1; = 0.9;
trainParam.goal = 1.0E-05;
net.trainParam.epochs = 500;

I find that  the training is very slow.
In fact it is really slow after the first 300 epochs.
I have about 70,000 sets of input patterns.
I did not know what value of learning rate to use.
I tried values from 0.05 to 20.0 but there was not
much improvement. My goal (trainParam.goal) was
0.00001 but the training is not going below 0.08
for some reason.

I am using Resilient Backpropagation (trainrp)
because "trainlm" is killing me. It takes too much
memory. I tried reducing memory using the option

This option worked but again I got stuck at 0.08
(mse) instead of going to 0.0001.

Using the values of weights for this level of training,
I tried to see if my outputs match my targets.

I did :

The output predicted is very bad. Obviously the network
was not properly trained.

Please help!


D. Thanar

2. Epson Perfection 600

3. help HELP HELP!!! please help...

4. CommandBarButton.State = MsoButtonState.msoButtonDown causes exception with Commands buttons. Known issue?

5. Introducing a new Data Parsing Toolbox.

6. Procmail + Qmail = piles of hair on the floor around my desk

7. Parse data to .m-file through a shell script

8. Compression with a fixed dictionary

9. !!!Parse and convert data!!!

10. load data and parse

11. How do I parse a data file that I imported in matlab.

12. Need help plotting 3-d data please

13. Please help with reading a data file